All work
LOG-03Shipped2026
GenAI Evaluation Kit: measuring whether a conversational agent actually works
The evaluation method from production, packaged: three quality pillars, 18 LLM-as-judge prompts, a golden-dataset template and a scoring dashboard.
- Context
- Digital product · sold to teams running GenAI assistants
- Role
- Author: method, prompts, templates
- Shows
- Eval mindset: golden datasets, judge calibration, quality gates before real traffic.
- 18
- LLM-as-judge prompts with structured JSON output
- 3
- pillars: answer reliability, tone, orchestration & flow
- 4
- go-live stages: lab, shadow, pilot, rollout
Problem
Teams put a GenAI assistant in production, management sees handle times drop, and nobody can answer the real questions. How many answers contain invented facts? Is the tone right with an angry customer? When the bot cannot help, is the fallback useful or a dead end? CSAT tells you something is wrong. It does not tell you what.
The method
- Three pillars. Answer reliability, tone of voice, orchestration & flow. Each has its own judge prompts and scoring scale.
- Golden dataset first. Built from real conversations with a deliberate 40/40/20 split between standard, tricky and adversarial cases.
- Calibrate the judge. An LLM-as-judge is only useful once it agrees with human reviewers on a sample. The kit includes the calibration loop.
- Gate the rollout. Lab → shadow → pilot → rollout, with explicit pass criteria at each step.
What I learned packaging it
Writing the method down for strangers forced every implicit rule from production into the open. Half of the value is the 22 worked examples in the dataset template: people copy examples, not frameworks.