From NLU to GenAI: What Actually Breaks When You Migrate an Enterprise Voicebot
The technology migration was the easy part. What broke was everything around it — evaluation criteria, KPIs, team workflows, and stakeholder expectations.
The Migration Nobody Warns You About
When we decided to migrate our enterprise digital assistant — tens of thousands of daily conversations, regulated environment, multi-language — from traditional NLU to a hybrid GenAI architecture, I thought the hardest part would be the technology. I was wrong by a wide margin.
The model swap took weeks. The organizational transformation around it took months.
The Old World: Intent Classification Paradise
Our NLU-based system was a beast. Hundreds of intents, thousands of utterance patterns, meticulously maintained by a team of conversation designers. It worked — until it didn't.
The fundamental problem with intent classification is rigidity. A customer says "I need to change my address but also I have a question about my last bill" and the NLU picks one intent. The other gets dropped. The customer repeats themselves. The bot asks them to "please choose one of the following options." The customer hangs up.
We had ~15% of conversations failing because real humans don't speak in clean, single-intent utterances. They ramble, they change topic, they interrupt themselves. NLU was designed for a world where humans cooperate with machines. They don't.
The Promise
GenAI offered something irresistible: flexible understanding. No more maintaining utterance patterns. No more "sorry, I didn't understand" on perfectly reasonable requests that just didn't match a predefined pattern.
The demos were convincing. The pilot results were promising. So we pulled the trigger.
What Actually Broke
1. Every KPI Became Meaningless
Our entire measurement framework was built for deterministic systems. Non-Handled Conversations (NHC)? The concept doesn't map cleanly when the bot can always generate some response. Containment rate? What counts as "contained" when the bot provides a conversational answer that's technically relevant but doesn't actually solve the problem?
We spent six weeks redefining every metric. The new framework measures task completion (did the customer achieve their goal?), conversation quality (scored by LLM-as-Judge), and customer effort (how many turns to resolution?). Completely different philosophy.
2. The Test Framework Died Overnight
You can unit-test an intent classifier. Input: "I want to cancel my subscription." Expected output: intent cancel_subscription, confidence > 0.85. Clean, deterministic, automatable.
You cannot unit-test an LLM the same way. The same input can produce different outputs. "Correct" isn't binary — it's a spectrum. We had to build an entirely new evaluation approach using LLM-as-Judge and golden datasets built from real production conversations.
Our QA team went from "run the test suite" to "evaluate conversation quality across multiple dimensions." It's a different skillset. Some people thrived. Others struggled. Nobody was prepared.
3. Vendor Contracts Didn't Fit
Our SLAs were written for deterministic systems. "99.5% intent classification accuracy" is measurable and contractual. What's the equivalent for a generative system? "99.5% of responses are good"? Good according to whom?
We spent three months renegotiating contracts with our technology partners. The legal team had never seen an SLA for a probabilistic system. Neither had the vendors, frankly.
4. The Operations Team Panicked
Our operations team was used to flowcharts. They could look at a conversation flow diagram and predict exactly what the bot would say in any situation. With GenAI, they couldn't.
"What if it says something wrong about a financial product?" "What if it gives medical advice?" "What if it insults a customer?"
These aren't hypothetical concerns in a regulated environment. We had to build guardrails, implement real-time content filtering, and create an incident response process for AI-generated content. The ops team needed training, not just on the technology, but on the new mental model of managing a probabilistic system.
The Hybrid Approach
We didn't go full GenAI. That would have been reckless.
Our architecture splits conversations into deterministic and generative flows:
- Deterministic: Authentication, transactions, anything involving money or personal data changes. These stay on hard-coded logic. No creativity needed, no risk tolerated.
- Generative: FAQ handling, troubleshooting, triage, conversational engagement. These benefit from flexibility and handle the long tail of customer requests.
- Orchestration layer: The routing logic between the two. This is where the real complexity lives — deciding in real-time whether a customer request needs the safety of deterministic handling or the flexibility of generative AI.
The orchestration layer is the most underestimated component. It's not glamorous, but it's where the system succeeds or fails.
The Real Lesson
The migration isn't a technology project. It's an organizational transformation that happens to involve technology.
Your measurement framework changes. Your testing methodology changes. Your vendor relationships change. Your team's skills need to evolve. Your stakeholders' expectations need to be managed.
If you're planning a similar migration, budget 30% of the effort for technology and 70% for everything else. You'll still underestimate the "everything else."
Planning an NLU-to-GenAI migration? I've done it at scale. Get in touch and let's map out your approach.