Voice Loop Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Voice Loop Engine is an on-device conversation system that handles the complete lifecycle of a voice-based interaction. It replaces traditional server-side pipelines by performing Voice Activity Detection (VAD), Speech-to-Text (STT), and Text-to-Speech (TTS) locally on the user's device. This architecture enables low-latency responses and local "barge-in" decisions without requiring a round-trip for audio processing.
Sources: apps/web/src/lib/live/voiceEngine.ts:7-12
Architecture and Core Components
The engine is built around the VoiceEngine class, which orchestrates several specialized modules to manage the audio loop.
Primary Components
| Component | Responsibility |
|---|---|
| MicVAD | Silero VAD (v5) segments user speech and detects start/end of utterances. |
| AudioPlayer | Manages audio playback, resumes contexts, and provides frequency band data. |
| SentenceChunker | Breaks LLM text deltas into stable-length fragments for TTS. |
| Worker Models | Whisper (STT) and Kokoro (TTS) running via WebGPU/WASM in a Web Worker. |
Sources: apps/web/src/lib/live/voiceEngine.ts:24-30, apps/web/src/lib/live/models.worker.ts:1-10
Engine Phases
The system transitions through four distinct states during a conversation:
- Idle: Waiting for user speech.
- Listening: Actively capturing and transcribing user audio.
- Thinking: User turn is complete; waiting for LLM response or tool results.
- Speaking: Transcribing LLM response deltas into audio and playing back.
Sources: apps/web/src/lib/live/voiceEngine.ts:13
Technical Workflow
The following diagram illustrates the data flow from initial microphone capture through to the final agent response.
The engine uses a local Decision path for "barge-in," where user speech during the speaking phase triggers an immediate halt of the AudioPlayer and an increment of the internal epoch to invalidate stale TTS jobs.
Sources: apps/web/src/lib/live/voiceEngine.ts:98-103, apps/web/src/lib/live/voiceEngine.ts:247-251
User Speech Processing
Voice Activity Detection (VAD)
The engine utilizes MicVAD with specific thresholds to balance sensitivity and noise rejection:
- Positive Threshold: 0.5 (for soft speech and fast barge-in).
- Redemption: 350ms (silence buffer before ending a turn).
- Noise Floor: A dynamic value learned while idle to prevent background noise from triggering turns.
Sources: apps/web/src/lib/live/voiceEngine.ts:52-65, apps/web/src/lib/live/voiceEngine.ts:109-114
Turn Detection and "Mid-Thought" Logic
To prevent cutting off users during natural pauses, the engine employs a "mid-thought" check.
- If the transcribed text ends with trailing filler words (e.g., "to", "the", "and") or the semantic model (
Smart-Turn) indicates the thought is incomplete, the engine holds the audio in apendingbuffer for up to 900ms. - isJunk Filtering: Whisper silence-hallucinations (e.g., "thanks for watching") are automatically dropped.
Sources: apps/web/src/lib/live/voiceEngine.ts:173-182, apps/web/src/lib/live/voiceText.ts:23-35
Agent Response and Text Processing
The engine processes incoming text deltas from the LLM using a SentenceChunker to ensure stable audio timbre and pacing.
Sentence Chunking Logic
- MIN_TTS_CHARS (40): Minimum characters required to send a fragment to the Kokoro TTS model to avoid rate/timbre instability.
- FIRST_TTS_CHARS (28): A lower bar for the very first chunk of a response to reduce perceived latency.
- Clause Boundaries: The first chunk can be released early if it hits a natural boundary like a comma or semicolon.
Sources: apps/web/src/lib/live/voiceText.ts:7-21, apps/web/src/lib/live/voiceText.ts:98-115
Audio Playback and Captions
Audio is enqueued into a ttsChain. The AudioPlayer uses an epoch system to handle barge-in.
- When an audio chunk starts playing,
onAgentTextis triggered to reveal the transcript word-by-word in sync with the audio. - Markdown symbols, citations, and photo-narration phrases are stripped before being sent to TTS.
Sources: apps/web/src/lib/live/voiceEngine.ts:219-245, apps/web/src/lib/live/voiceText.ts:47-65
Sequence of a Live Interaction
This sequence diagram depicts the interaction between the UI, the Voice Engine, and the local models.
Summary
The Voice Loop Engine provides a high-performance, low-latency framework for multimodal AI interaction. By processing VAD, STT, and TTS on the client, the system remains responsive even with network latency, allowing for natural features like barge-in and real-time audio visualization via octave bands.
Sources: apps/web/src/lib/live/voiceEngine.ts:7-12, apps/web/src/lib/live/voiceEngine.ts:275-285