Migrating a 70K-conversations-a-day voicebot from NLU to hybrid GenAI
Moving a regulated, high-volume customer-service assistant off intent classification without breaking what customers and the business rely on.
- Context
- Tier-1 Italian enterprise · regulated · voice + chat
- Role
- GenAI specialist on the deployment team: architecture, evaluation, KPIs, rollout
- Shows
- End-to-end ownership of a production migration: architecture split, evals, KPI redesign, vendor SLAs, change management.
- 70K+
- conversations a day in scope
- ~15%
- conversations lost to multi-intent utterances before the migration
- 6 wks
- to rebuild the KPI framework for a probabilistic system
Problem
The assistant ran on classic intent classification: hundreds of intents, thousands of utterance patterns, a team of conversation designers keeping it alive. It worked until customers spoke like humans. "I need to change my address, and also a question about my last bill" gets one intent. The other is dropped, the customer repeats themselves, and often hangs up. Roughly 15% of conversations failed this way.
Constraints
- Regulated domain. Anything touching money, identity or personal-data changes cannot be improvised by a model.
- Volume. Tens of thousands of conversations a day: a 1% regression is hundreds of unhappy customers before lunch.
- Contracts and KPIs written for deterministic systems. "99.5% intent accuracy" has no generative equivalent.
Architecture
We did not go full GenAI. The journey is split by risk, with an orchestration layer deciding in real time which side a request belongs to.
┌──────────────────────────┐
customer ───▶ │ orchestration / routing │ ◀── context, auth state, risk policy
└────────────┬─────────────┘
┌───────────────┴────────────────┐
▼ ▼
┌─────────────────────┐ ┌──────────────────────┐
│ DETERMINISTIC flows │ │ GENERATIVE flows │
│ auth · payments · │ │ FAQ · triage · │
│ personal-data edits │ │ troubleshooting │
└──────────┬──────────┘ └──────────┬───────────┘
└──────────────┬─────────────────┘
▼
human handoff with summary
The orchestration layer is the least glamorous component and the one where the system succeeds or fails.
What I did
- Redefined the KPIs. Containment and "non-handled conversations" stop meaning much when a bot can always say something. We moved to task completion, conversation quality scored by a calibrated LLM-as-judge, and customer effort (turns to resolution).
- Replaced the test suite. Intent unit tests became golden datasets built from real production transcripts, evaluated on multiple dimensions.
- Staged the rollout. Lab → shadow → pilot → rollout. In shadow mode the system answered silently on 500+ sampled conversations a day for one to two weeks before it spoke to a customer.
- Rewrote the vendor SLAs for a probabilistic system, together with legal and the technology partners.
What broke
- Every KPI. Six weeks to rebuild the measurement framework.
- The QA workflow. "Run the suite" became "judge quality across dimensions". Different skill set; not everyone made the jump.
- Contracts. Three months of renegotiation: nobody on either side had seen an SLA for a generative system.
- Operations' trust. Ops could predict a flowchart; they could not predict a model. Guardrails, real-time filtering and an incident process for AI-generated content came before trust did.
What I would do differently
Budget 30% of the effort for technology and 70% for everything around it, then assume you still underestimated the 70%. Start the KPI and SLA redesign on day one, not after the pilot.