Lab → Shadow → Pilot → Rollout: How We Test GenAI Before It Talks to Real Customers
You can't A/B test a digital assistant that handles financial inquiries with 'let's see what happens.' Here's the 4-stage framework I use.
Why Traditional QA Doesn't Work
Traditional software testing is deterministic. Input A produces Output B. If it doesn't, there's a bug. Fix the bug, run the test again, move on.
GenAI systems are probabilistic. The same input can produce different outputs. "Correct" is a spectrum, not a boolean. And when your system handles financial service inquiries for thousands of people daily, "let's see what happens" is not an acceptable test strategy.
I designed this 4-stage framework out of necessity. Our compliance team wouldn't approve a production deployment without structured validation. Our operations team wouldn't support a system they couldn't predict. And our customers wouldn't tolerate a bot that hallucinates about their pension.
Stage 1: Lab — Controlled Testing with Golden Datasets
Before any GenAI system sees real traffic, it faces the golden dataset.
A golden dataset is a curated collection of ~200 representative conversations, built from real production verbatims. Not synthetic data — actual conversations recorded, transcribed, and annotated by hand. You go to the operations floor, you listen to calls, you transcribe them. There's no shortcut.
Each conversation is scored by an LLM-as-Judge across multiple dimensions:
- Factual accuracy: Does the response contain correct information?
- Tone of voice compliance: Does it match our brand guidelines?
- Resolution completeness: Does it actually solve the customer's problem?
- Escalation appropriateness: Does it know when to hand off to a human?
The Lab stage catches ~80% of issues before anything goes live. It's the cheapest place to fail. If your system can't handle curated examples correctly, it definitely can't handle the chaos of real conversations.
Stage 2: Shadow — Listen Without Speaking
The shadow stage is where the GenAI system listens to real conversations but doesn't respond. It generates what it would have said, and we compare that to what the existing system (or human agent) actually said.
Zero risk, maximum learning.
This stage reveals patterns that golden datasets miss. Real traffic is messier — customers with background noise, mid-sentence topic changes, regional dialects, and the occasional person who calls to chat about the weather.
We run shadow mode for 1-2 weeks, sampling 500+ conversations daily. The LLM-as-Judge evaluates every shadow response. We look for systematic failures: categories of questions the system handles poorly, tone inconsistencies, and hallucination patterns.
Shadow mode is also where the operations team builds confidence. They can see exactly what the AI would say, compare it to what the human said, and start forming expectations about the system's behavior.
Stage 3: Pilot — Real Traffic, Training Wheels
Pilot means real traffic. Real customers talking to the real GenAI system.
But with guardrails:
- Small percentage: 5-10% of traffic, randomly sampled
- Real-time monitoring: Dashboard showing every conversation as it happens
- Instant rollback: One button to route all traffic back to the previous system
- Daily human review: A team samples 50+ conversations per day, scoring them manually
Metrics we track during pilot:
- Resolution rate (did the customer achieve their goal?)
- Customer satisfaction (post-call survey)
- Escalation rate (how often does AI hand off to human?)
- Hallucination incidents (factually incorrect statements)
- Average handling time
The pilot stage typically runs 2-3 weeks. If metrics are stable and improving, we proceed. If there's any degradation in customer satisfaction or a spike in hallucination incidents, we pause and investigate.
Stage 4: Rollout — Never Go to 100% in One Step
Rollout is gradual. Always.
Our cadence: 5% → 15% → 30% → 50% → 100%, with at least 3-5 days at each level. At every stage, we compare metrics against the baseline. Any significant regression triggers a pause.
The temptation to accelerate is enormous. Stakeholders want to see ROI. The team is excited. "It's working great at 30%, let's just go to 100%!"
Don't. Scale reveals issues that small samples hide. A rare failure mode that occurs in 0.1% of conversations is invisible at 5% traffic (maybe 3 occurrences per day) but becomes 70 angry customers at 100%. You want to find that failure mode at 30%, not at 100%.
The LLM-as-Judge Approach
Manual conversation review doesn't scale. You can't have humans listen to tens of thousands of conversations daily. But you can have an LLM evaluate every single one.
We use a separate LLM (typically a stronger model than the production one) as an automated judge. It scores each conversation on our evaluation dimensions using structured rubrics. Vague criteria like "was the answer good?" produce unreliable scores. Specific criteria like "did the response correctly identify the customer's request type?" produce consistent, actionable data.
The judge model gets calibrated against human evaluators. We measure inter-rater reliability between human scores and LLM scores. When they align above 85%, we trust the automated scoring for the full volume and reserve human review for flagged conversations.
The Framework Seems Slow. That's the Point.
Four stages. Multiple weeks. Gradual rollout. In a world obsessed with shipping fast, this feels painfully slow.
But speed in testing is how you get hallucinations in production. And in a regulated environment, one hallucination about a financial product can trigger a compliance review that freezes your entire AI program for months.
The framework isn't slow. It's the fastest path to a deployment you can actually sustain.
I packaged this evaluation methodology into the GenAI Evaluation Kit — templates, rubrics, and scoring frameworks for teams doing this themselves.