On-Device Voice Models
Relevant source files
The following files were used as context for generating this wiki page:
The On-Device Voice Models system provides a fully local pipeline for speech-to-text (STT), text-to-speech (TTS), and semantic end-of-turn detection. By leveraging WebGPU and WASM via transformers.js and ONNX Runtime, the application processes voice data directly in the browser. This architecture ensures low latency for voice interactions and maintains privacy as no audio data for these specific tasks is sent to external servers.
This system is primarily orchestrated by the VoiceEngine which coordinates with a background Worker to perform heavy computational tasks without blocking the main UI thread. Sources: apps/web/src/lib/live/models.worker.ts:1-10, apps/web/src/lib/live/voiceEngine.ts:8-15
System Architecture
The architecture follows a "Thick Client" pattern where the frontend handles the immediate voice loop: Voice Activity Detection (VAD), transcription, and speech synthesis.
High-Level Component Interaction
The following diagram illustrates how the main thread interacts with the background worker and the physical hardware.
The VoiceEngine captures audio from the microphone and sends it to the Models Facade, which communicates via postMessage to the dedicated models.worker.ts. Sources: apps/web/src/lib/live/useLiveSession.ts:31-45, apps/web/src/lib/live/models.ts:3-15
Core Models and Capabilities
The system utilizes three primary models, optimized for either WebGPU (fast) or WASM (fallback) execution.
| Model Role | Technology | Implementation | Target Device |
|---|---|---|---|
| STT (Speech Recognition) | Whisper | whisper-base.en / whisper-tiny.en | WebGPU / WASM |
| TTS (Voice Synthesis) | Kokoro | Kokoro-82M-v1.0-ONNX | WebGPU / WASM |
| Turn Detection | Smart-Turn v3 | smart-turn-v3.2-cpu.onnx | WASM (CPU) |
Sources: apps/web/src/lib/live/models.worker.ts:25-33, apps/web/src/lib/live/models.ts:18-24
Model Loading and Caching
Models are downloaded from the Hugging Face hub on the first request and stored using the browser's Cache API.
- Persistent Cache: The system uses
localStorageto track if models are already cached (takt-live-models-ready-v1). - Device Tiering: The system automatically detects WebGPU availability to decide between
fp32(high performance) andq8(quantized) model variants.
Sources: apps/web/src/lib/live/models.ts:27-35, apps/web/src/lib/live/models.worker.ts:14-22
Voice Processing Pipeline
The voice loop manages the transition between different states of conversation.
Execution Flow
- VAD Trigger:
MicVADdetects speech onset. - Transcription:
Whisperprovides partial and final transcripts. - Semantic Check:
Smart-Turndetermines if the user has finished their thought. - Speech Synthesis:
Kokorogenerates audio for the assistant's reply.
Text and Audio Refinement
Before text is synthesized into speech, it undergoes "scrubbing" to ensure a natural voice experience.
Sentence Chunking
The SentenceChunker prevents the assistant from speaking tiny fragments that might cause timbre or rate fluctuations. It holds short sentences until they meet the MIN_TTS_CHARS threshold (approximately 35 characters) or until the turn is flushed. Sources: apps/web/src/lib/live/voiceText.ts:106-125
Content Scrubbing
Spoken text is cleaned using stripMarkdown to remove:
- Markdown symbols (e.g.,
**,[p.18]). - Visual narration phrases (e.g., "In the image I see..." becomes "here I see...").
- Provider-specific control tokens (e.g., MiniMax's
[e[).
Sources: apps/web/src/lib/live/voiceText.ts:80-100.%0A%0ASources%3A%20%5Bapps/web/src/lib/live/voiceText.ts%3A80-100)
User Interface and Feedback
The system provides real-time feedback during the model download and active call phases.
- Download Progress: Managed by
DownloadProgressin the UI, showing per-model percentages and MB loaded. - Visual Spectrum: The
VoiceEngineprovides frequency band energy data viamicBands()andagentBands(), derived from anAnalyserNodeto drive UI visualizations (e.g., the "orb").
Sources: apps/web/src/components/live/LiveStage.tsx:18-45, apps/web/src/lib/live/voiceEngine.ts:219-230
Technical Constraints and Optimizations
- Serialization: To prevent GPU contention, the worker serializes all inference jobs using a promise chain.
- Smart-Turn Window: The semantic turn model specifically analyzes the last 8 seconds of audio (8 * 16,000 samples).
- Concurrency:
onnxruntime-webis configured withnumThreads = 1to avoid requirements for cross-origin isolation.
Sources: apps/web/src/lib/live/models.worker.ts:18-40, apps/web/src/lib/live/models.worker.ts:50-55
The On-Device Voice Models system integrates low-level audio processing with modern browser AI capabilities to deliver a responsive, private, and robust voice interface for the Nudge tutor.