Voice Mode & TTS/STT
Relevant source files
The following files were used as context for generating this wiki page:
The Voice Mode & TTS/STT system in Onyx provides real-time, interactive audio capabilities, allowing users to interact with the platform using spoken language. It encompasses Text-to-Speech (TTS) for synthesizing agent responses and Speech-to-Text (STT) for transcribing user input. The system is designed for low-latency performance through streaming architectures, using WebSockets for real-time audio delivery and specialized buffering strategies to minimize perceived latency.
This module integrates with external providers such as OpenAI, ElevenLabs, and Azure Speech Services to handle the intensive compute requirements of high-quality voice synthesis and recognition. Within the frontend, it provides synchronized text-to-audio reveal patterns, ensuring that the displayed response matches the spoken output.
Sources: web/src/providers/VoiceModeProvider.tsx:1-125, web/src/app/admin/configuration/voice/VoiceProviderSetupModal.tsx:45-75
Architecture and Data Flow
The voice system utilizes a client-server architecture where the frontend manages audio hardware, playback buffers, and state, while the backend coordinates with external AI providers.
Text-to-Speech (TTS) Pipeline
The TTS system employs a streaming WebSocket connection to deliver audio chunks as they are generated. The frontend uses a VoiceModeProvider to manage this connection and handle audio playback via the MediaSource API.
The system uses a "fast-start" mechanism where the first TTS chunk is sent as soon as a minimum character threshold (typically 20 characters) is reached, rather than waiting for a full sentence boundary. This reduces perceived latency to approximately 200ms after text generation begins.
Sources: web/src/providers/VoiceModeProvider.tsx:390-450, web/src/providers/VoiceModeProvider.tsx:640-675
Implementation Details
The following table summarizes the core components and their roles in the voice architecture:
| Component | Responsibility |
|---|---|
VoiceModeProvider | Manages global audio state, WebSocket lifecycle, and MediaSource buffering. |
VoiceProviderSetupModal | Admin interface for configuring API keys and model selections for OpenAI, Azure, and ElevenLabs. |
MediaSource API | Used by the frontend to play back binary audio chunks arriving via WebSocket without interrupting the stream. |
WS_TOKEN_ENDPOINT | Provides a short-lived (60s TTL) single-use token for authenticating WebSocket connections. |
Sources: web/src/providers/VoiceModeProvider.tsx:40-120, web/src/app/admin/configuration/voice/VoiceProviderSetupModal.tsx:100-150
Configuration and Providers
Onyx supports multiple external providers for voice services, each requiring specific configuration through the Admin UI.
Supported Providers and Models
The system currently integrates with three primary providers for TTS and STT:
| Provider | Supported Modes | Configuration Requirements |
|---|---|---|
| OpenAI | TTS, STT | API Key, Model ID (e.g., tts-1, whisper-1) |
| ElevenLabs | TTS | API Key, Voice ID |
| Azure | TTS, STT | API Key, Target URI (Endpoint/Region) |
Admin Setup Logic
When configuring a provider, the system performs a validation check. For OpenAI, the platform supports "API Key Reuse," allowing administrators to select existing keys used for LLM providers.
Frontend Audio Management
The frontend implementation focuses on state synchronization between the generated text and the synthesized audio.
Progressive Text Reveal
A critical feature of the VoiceModeProvider is synchronized text reveal. The system tracks audioProgress (calculated from audio.currentTime and estimated duration) and uses an animation frame loop to reveal characters progressively. This prevents the text from outrunning the audio.
- Lead Time: The text reveal leads the audio playback by approximately 0.28 seconds to ensure smooth visual flow.
- Smoothing: A maximum of 8 characters can be revealed per animation frame to prevent sudden "jumps" in text display.
- Cleanup: Text is cleaned of Markdown formatting (removing bold markers, links, etc.) before being sent for synthesis to ensure the agent speaks naturally.
Sources: web/src/providers/VoiceModeProvider.tsx:125-150, web/src/providers/VoiceModeProvider.tsx:755-800
State Tracking
The VoiceModeProvider exposes several reactive states for the UI:
isTTSPlaying: True when audio is actively coming out of the speakers.isTTSLoading: True when the system is awaiting the first audio chunks from the server.manualStopCount: Incremented when a user manually interrupts playback, allowing other components to react to the cessation of audio.isAwaitingAutoPlaybackStart: A "thinking" state used to signal that text has been queued but audio hasn't yet reached the buffer.
Sources: web/src/providers/VoiceModeProvider.tsx:70-120, web/src/providers/VoiceModeProvider.tsx:500-530
Prompts and Reminders
The behavior of the Voice Mode is influenced by specialized system prompts and reminders that ensure the LLM understands its role in an audio-first context.
Voice-Specific Instructions
When certain tools (like image generation) are used during a voice session, reminders are appended to the context to ensure the agent describes its actions for a listener.
IMAGE_GEN_REMINDER = """
Very briefly describe the image(s) generated. Do not include any links or attachments.
""".strip()
Sources: backend/onyx/prompts/chat_prompts.py:100-110
Text Preparation
Before synthesis, the system applies cleanTextForTTS to the LLM response. This logic removes:
- Markdown Bold/Italic:
**text**becomestext. - Links:
[Onyx](https://onyx.app)becomesOnyx. - Code Markers: Backticks are stripped.
- Whitespace: Newlines are replaced with spaces to create a continuous stream for the synthesizer.
Sources: web/src/providers/VoiceModeProvider.tsx:135-155
Summary
The Voice Mode & TTS/STT system provides a low-latency, streaming audio interface for Onyx. By leveraging WebSockets and the frontend MediaSource API, it achieves near-instantaneous synthesis. The architecture is modular, allowing for easy expansion to new providers while maintaining strict control over audio-text synchronization and Markdown-to-speech cleaning.
Sources: web/src/providers/VoiceModeProvider.tsx, backend/onyx/chat/README.md