Real-Time Voice Agent
Design a low-latency speech-to-speech conversational agent with streaming ASR, LLM, TTS, and barge-in support targeting sub-1s perceived response time.
Last updated: September 2026
A low-latency speech-to-speech conversational agent - user speaks, agent listens, responds in natural spoken language with minimal delay and barge-in support. Every stage streams; no stage waits for the previous to fully finish. The core latency technique is sentence-boundary streaming: dispatch each complete sentence to TTS immediately while the LLM generates the next sentence - LLM and TTS overlap, achieving sub-1s perceived latency despite multiple pipeline stages.
“Stream everything. Sentence boundary is the unit of dispatch. Speculative processing beats waiting - start LLM on interim ASR, don't wait for final. Barge-in is a first-class feature, not an edge case.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| Latency budget? (< 500ms? < 1s?) | Drives almost every architectural choice |
| Domain-specific or open-ended? | Specific = smaller, faster, fine-tuned models |
| Barge-in needed? (user interrupts agent) | Significantly harder - explicit state machine required |
| Noise environment? (office vs. call center) | ASR model selection + noise cancellation |
| Transport? (browser WebRTC, phone SIP, mobile) | Codec, echo cancellation, signaling approach |
| How many concurrent sessions? | Infrastructure topology |
| Tool calls needed? (CRM, calendar lookups) | Adds latency - need filler strategy |
Architecture (Draw This)
User speaks → [1] Audio Capture (WebRTC/SIP) - echo cancellation at client → [2] VAD (is user speaking? did they stop? barge-in?) → [3] ASR (streaming: interim words → final transcript) → [4] NLU + Conversation State (intent, entities, turn history in Redis) → [5] LLM (stream tokens → sentence boundary detected → dispatch to TTS) → [6] TTS (stream audio → first byte < 200ms) → [7] Playback + Barge-in Control User hears agent Core insight: every stage streams. No stage waits for the previous to fully finish.
Latency Budget (Memorize This)
| Stage | Target |
|---|---|
| Audio → server (network) | 20-50ms |
| VAD endpoint detection | 300-500ms (silence window) |
| ASR final transcript | 50-100ms after endpoint |
| LLM first token | 150-300ms |
| TTS first audio byte | 150-200ms |
| Audio → client (network) | 20-50ms |
| Total (first word heard) | ~700ms-1.2s |
Key Components (30-second pitch each)
- 1VAD - Voice Activity Detection - End-of-turn detection: 500ms silence after speech = user done speaking. Barge-in: run VAD continuously even while agent is speaking. User speech detected → stop TTS immediately + clear audio buffer + switch to ASR mode. Use neural VAD (Silero VAD) not energy threshold - energy fails in noise.
- 2ASR - Streaming Speech-to-Text - Google Cloud Speech-to-Text Streaming / Deepgram / AssemblyAI. Emits interim (unstable) + final (committed) results. Speculative prefill: feed interim results to LLM early to start planning. Correct on final. Saves 200-400ms perceived latency.
- 3LLM - The Brain - Use fast models: Gemini Flash, Claude Haiku - not GPT-4-class. Latency > quality for voice. Stream tokens. Parse sentence boundaries. Dispatch each complete sentence to TTS immediately. System prompt must enforce voice style: short sentences, no markdown, no lists, contractions, natural speech.
- 4TTS - Text-to-Speech - Google Cloud TTS Neural2 / ElevenLabs / Azure Neural TTS. Sentence-level dispatching: TTS for sentence 1 starts while LLM generates sentence 2 - this is the biggest latency win. Target first-byte < 200ms. Buffer 100-200ms before playback to smooth jitter.
- 5Agent State Machine - States: IDLE → LISTENING → TRANSCRIBING → THINKING → SPEAKING. Barge-in interrupt must complete in ~50ms to feel natural. Add 200ms silence padding after agent speech before re-enabling VAD to prevent trailing audio triggering false barge-in.
The #1 Latency Technique - Sentence-Boundary Streaming
User speech ends ASR → final transcript LLM streams: "Sure, [.]" → dispatch sentence 1 to TTS immediately TTS plays sentence 1 while LLM generates: "Let me check that. [.]" → dispatch sentence 2 User hears sentence 1 while sentence 2 is being synthesized LLM and TTS overlap - this is how you achieve sub-1s perceived latency despite multiple stages.
Infrastructure Requirements
| Requirement | Why |
|---|---|
| Co-locate ASR + LLM + TTS in same region | Each network hop = 10-50ms penalty |
| gRPC bidirectional streaming between services | Not HTTP - connection setup overhead is too high on hot path |
| GPU required for LLM | CPU inference is 10-100x slower |
| Session-affinity load balancing | Each session is stateful long-lived connection |
3 Biggest Risks
- 1VAD tuning - Too aggressive clips users, too conservative feels sluggish. Requires empirical tuning on real audio samples from the target noise environment.
- 2LLM latency - Large models cannot meet sub-1s budgets. Model selection is critical; fine-tuned small model beats general large model for voice in narrow domains.
- 3Barge-in false positives in noisy environments - Mitigated by echo cancellation + confidence threshold + 50ms sustained speech confirmation window before triggering barge-in.
Google Stack: Cloud Speech-to-Text Streaming (ASR) → Gemini Flash (LLM) → Cloud TTS Neural2 (synthesis) → Chirp (on-device ASR option) → Redis Memorystore (session state) → GKE with session-affinity (scaling)
Azure Stack: Azure Communication Services (WebRTC/SIP transport, echo cancellation) → Azure Speech Service Streaming STT (real-time ASR, WebSocket v2, 180-250ms P50 first-partial) → Azure OpenAI GPT-4o Realtime (native audio-in / audio-out, lowest latency path) → Azure AI Speech TTS Neural (first-byte <200ms) → Azure Cache for Redis (session state) → AKS with session-affinity (scaling)
AWS Stack: Amazon Chime SDK / Amazon Connect (WebRTC/SIP transport) → Amazon Transcribe Streaming (ASR, WebSocket) → Amazon Nova 2 Sonic (end-to-end voice model: native STT + LLM + TTS in one call, lowest latency on AWS) → Amazon Polly Neural TTS (fallback TTS) → ElastiCache Redis (session state) → Amazon EKS with sticky sessions (scaling)
Other Options: Deepgram Nova-3 (ASR, <150ms latency) + OpenAI Realtime API + ElevenLabs TTS (best-in-class latency, provider-agnostic) | Twilio Voice + Deepgram + Claude Haiku (telephony-native path) | LiveKit + Silero VAD + Whisper + local LLM (fully open-source, on-prem deployment)
Frequently asked questions
How do you get below 500ms?
Three levers: (1) speculative LLM prefill on interim ASR, (2) co-locate ASR + LLM on same machine (removes one hop), (3) smaller domain-fine-tuned LLM. Each saves 100-200ms. Achievable for narrow-domain agents.
How do you tell natural pause (comma) from end of turn?
VAD silence threshold tuning - mid-sentence pause < 300ms, end-of-turn > 500ms. Also use ASR completeness signals and prosodic cues (pitch/energy drop). Wrong tuning = clipping (too aggressive) or sluggish (too conservative).
User says 'um... uh...' - does VAD reset the silence timer?
Yes - the filler word problem. Fix: disfluency detection model that marks fillers and doesn't reset timer. Or ASR strips fillers before sending to LLM.
Barge-in triggered by background noise?
Echo cancellation removes agent's own voice. Noise gating + VAD confidence threshold prevent ambient noise triggering barge-in. Add 50ms sustained speech confirmation window before triggering.
User barges in during a tool call - what happens?
Cancel the tool call if possible (cancellation token). If already committed (API write), flag result as stale and don't surface in new turn.
LLM responses sound robotic when spoken aloud?
System prompt enforces voice style: short sentences, no markdown, contractions, natural transitions. Post-process output to strip any remaining markdown before TTS.
1 million concurrent sessions?
Session-affinity horizontal scaling. Shared GPU pools for LLM with strict per-session latency SLOs. Distribute across regions for telephony locality. At this scale: smaller models + response caching for frequent queries.