AI Voice Agents — Frequently Asked Questions
Answers on building conversational voice — the latency budget, barge-in, why recognition accuracy is the wrong metric, disclosure duties, and when voice is genuinely the right channel.
We have not built one. See AI Voice Agents for the full problem study and why. These answers are analysis, not experience of shipping a production voice system.
Feasibility#
Is the technology good enough now?#
The components are. Recognition, generation and synthesis are all strong individually. What is hard is integrating them within a conversational latency budget while handling interruption, background noise and recovery — a systems problem rather than a model problem.
That is why voice demos impress in a quiet room and disappoint on a real phone line.
What latency is acceptable?#
Comfortable human turn-taking sits in the region of a few hundred milliseconds. Past roughly a second, callers assume the line dropped and start talking again, which corrupts the turn.
The number that matters is measured end to end on a real telephone path including network — not in a browser demo on the same machine.
Where does the latency actually go?#
Across the whole chain, not one component: detecting that the speaker has finished, transcription, understanding and any retrieval or tool call, speech synthesis, and network in both directions.
This is why swapping in a faster model rarely rescues a slow agent. The budget is consumed by the chain.
What is barge-in and why does it matter?#
The ability for the caller to interrupt mid-sentence. Humans do this constantly. An agent that cannot be interrupted is experienced as rude, and callers talk over it anyway — leaving you with overlapping audio to resolve.
Get barge-in working before evaluating anything else. A system without it will not be judged on its accuracy.
Getting it wrong#
What is the most common mistake?#
Treating it as a speech problem rather than a conversation problem. Recognition accuracy is the easiest thing to measure and improve, so it absorbs the effort, while the rhythm of the exchange — which is what callers actually experience — receives none.
Two systems, one transcribing near-perfectly with awkward pauses and no interruption handling, the other less accurate but conversational: callers prefer the second, consistently.
What should we measure instead?#
Task completion rate and escalation rate — did the caller get what they rang for, and how often did a human have to finish it.
Word error rate belongs on an engineering dashboard, not in a decision about whether the system is working. It is possible to improve it measurably while completion falls, if the change added latency.
What breaks that never shows in testing?#
Background noise and second voices; names, addresses and reference numbers, which recognition handles worst and which most calls exist to exchange; accents and code-switching between languages mid-sentence; and silence, where a caller thinking is indistinguishable from a caller who has gone.
Any serious deployment needs a confirmation strategy for identifiers, and confirmation costs turns.
How should failure be handled?#
With a fast, obvious route to a human. When the agent mishears twice, most designs ask a third time; callers do not tolerate that.
The escalation path is the product, not a fallback. A voice agent without one converts a minor problem into a complaint.
Compliance and design#
Does the agent have to say it is AI?#
In many cases yes — EU transparency obligations for AI interaction have applied since 2 August 2026, and a voice agent presenting as a person is the clearest case they address.
Design disclosure in from the start rather than appending it. Which rules apply to you is a question for your own legal advisers.
What about recording calls?#
Voice is personal data, and recording carries specific consent requirements in many jurisdictions. Treat it as a design constraint from the beginning — retrofitting consent flows into a live call path is unpleasant.
Does tone matter?#
More than in text. Callers reaching support are frequently already frustrated, and cheerfulness in the face of a problem reads as indifference. The register that works in a chat widget does not transfer.
When to use it#
When is voice genuinely the right channel?#
When hands or eyes are busy — driving, warehouse, clinical, field work. For accessibility, where voice may be the usable interface. For high-volume, narrow, verifiable transactions with confirmation. And where the alternative is a long hold queue, provided escalation works.
When is it the wrong one?#
When a form would do. Voice is slower than typing for anything the caller can see and edit, and it is worse for anything requiring precision — long identifiers, addresses, amounts.
Novelty is not a use case. If the caller would rather have used a form, they will tell you by pressing zero.
What would you need before building one?#
A use case where voice genuinely beats the alternative; a latency budget measured on a real line; barge-in working; a confirmation strategy for identifiers; disclosure and consent designed in; a human escalation path faster than callers expect; and measurement of task completion rather than transcription accuracy.
We have not had a problem that justified it, which is why there is nothing here to demonstrate.