How We Evaluate LLMs in Production (Not Just Vibes)

The Vibes Problem

"It seems to work well." "The answers look good." "It feels better than the last model."

This is how most organizations evaluate their LLMs. I call it vibes-based evaluation, and it's how you end up deploying a model that works great on demos and fails on the 47th edge case your customer support team encounters every morning.

After years of deploying LLMs in production — and building the GenAI Evaluation Kit to systematize this process — I've developed a methodology that replaces gut feeling with data.

Why Benchmarks Aren't Enough

MMLU, HellaSwag, HumanEval — these benchmarks tell you something about a model's general capabilities. They tell you nothing about how it will perform on your specific use case.

Your customer service bot doesn't need to solve graduate-level physics. It needs to correctly identify whether a customer is asking about their invoice or their delivery, in Italian, with typos, at 2 AM.

Production evaluation is domain-specific, context-dependent, and ongoing. It's not a one-time test — it's a continuous practice.

The LLM-as-Judge Approach

The most scalable evaluation method I've found is using another LLM as a judge. Here's how it works:

You define evaluation criteria (accuracy, relevance, tone, safety)
You create a scoring rubric for each criterion (1-5 scale with specific descriptions)
You pass the original query, the model's response, and the rubric to a judge model
The judge scores each criterion and provides reasoning

This sounds circular — using AI to evaluate AI — but it works remarkably well when done right. The key is the rubric. Vague criteria like "is the answer good?" produce unreliable scores. Specific criteria like "does the response correctly identify the customer's intent as one of: billing, delivery, returns, other?" produce consistent, actionable scores.

I typically use a stronger model as the judge (GPT-4 class) to evaluate a faster, cheaper model in production (GPT-3.5 or Claude Haiku class). The economics work because you're judging a sample, not every interaction.

Golden Datasets: Your Ground Truth

Every evaluation system needs a ground truth to calibrate against. I build golden datasets — curated sets of query-response pairs that represent the full spectrum of expected interactions.

A good golden dataset has:

Happy path examples: The 80% of interactions that go smoothly
Edge cases: The 15% that require nuanced handling
Adversarial examples: The 5% that try to break the system (prompt injection, out-of-scope requests, offensive content)
Domain-specific traps: Queries that look simple but require domain expertise to answer correctly

I aim for 200-500 examples per use case. Fewer than that and you're not covering enough ground. More than that and maintenance becomes a burden.

The golden dataset isn't static. Every time you find a production failure, it becomes a new test case. Every time a customer complaint reveals a blind spot, it goes in the dataset. Over time, your golden dataset becomes your most valuable IP.

The Lab → Shadow → Pilot → Rollout Cycle

I never deploy a model change directly to production. Instead, I follow a four-stage cycle:

Lab (offline evaluation)

Run the candidate model against your golden dataset. Compare scores with the current production model. If the candidate doesn't beat the baseline on at least 90% of criteria, stop here.

Shadow (parallel evaluation)

Deploy the candidate model alongside the production model. Both receive the same real traffic, but only the production model's responses reach users. Compare their outputs on every interaction.

This stage catches issues that golden datasets miss — real-world traffic is always messier than curated examples.

Pilot (controlled exposure)

Route 5-10% of traffic to the candidate model. Monitor closely: response quality, latency, error rates, user satisfaction signals (did they ask to repeat? did they escalate to human?).

Rollout (gradual release)

If pilot metrics are positive, gradually increase traffic: 25%, 50%, 75%, 100%. At each stage, confirm that metrics hold. Keep the previous model available for instant rollback.

This cycle takes 2-4 weeks for a minor model update, 4-8 weeks for a major model change. It sounds slow, but I've seen "fast" deployments cause outages that took months to recover from.

Metrics That Actually Matter

Forget accuracy in isolation. These are the metrics I track:

Task completion rate: Did the user achieve what they came for?
Escalation rate: How often does the AI hand off to a human?
First-contact resolution: Was the issue resolved without follow-up?
User effort score: How many turns did the conversation take?
Containment rate: Of users who started with AI, how many stayed with AI?

Notice that none of these are about the model's intelligence. They're about the user's outcome. A "dumber" model with better conversation design often outperforms a "smarter" model with poor UX.

Building Your Own Evaluation Practice

If you're starting from zero:

Define 3-5 evaluation criteria specific to your use case
Build a golden dataset of 100 examples (start small, grow over time)
Set up automated scoring with LLM-as-Judge
Run evaluations on every model or prompt change
Track metrics over time — look for trends, not snapshots

The GenAI Evaluation Kit includes templates for all of this: scoring rubrics, golden dataset structure, judge prompts, and a metrics dashboard. It's the same methodology I use with enterprise clients, packaged for teams doing it themselves.

Need help setting up an evaluation practice for your team? Get in touch.