Voice Agent Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Voice Agent Pipeline orchestrates multiple AI primitives—Speech-to-Text (STT), Language Model (LLM), and Text-to-Speech (TTS)—into a unified, low-latency conversational interface. It manages the transition between listening, thinking, and speaking states, ensuring that on-device inference provides a natural user experience.
The pipeline exists as a core C++ component within runanywhere-commons and is exposed through various platform SDKs (Swift, Kotlin, React Native, Web, and Electron). It handles the complexities of audio buffer management, energy-based or model-based Voice Activity Detection (VAD), and the sequencing of inference tasks.
Sources: AGENTS.md, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:18-35
Architecture and Components
The Voice Agent Pipeline functions as a higher-level orchestrator that sits above individual AI engines. It utilizes a plugin registry to find and bind to specific backends for each required modality.
Pipeline Orchestration Flow
The following diagram illustrates the data flow from audio input to synthesized speech response.
The pipeline transitions through internal states to coordinate these components.
Sources: bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:145-165, bindings/electron/test/feature/voice.feature.test.ts:18-30
Core Modalities
The pipeline requires four primary components to be resident in memory for a successful session:
| Component | Description | Requirement |
|---|---|---|
| VAD | Voice Activity Detection; identifies speech start/end. | Mandatory |
| STT | Speech-to-Text; transcribes user utterances. | Mandatory |
| LLM | Large Language Model; generates text responses. | Mandatory |
| TTS | Text-to-Speech; synthesizes the response into audio. | Mandatory |
Sources: bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:448-460, bindings/electron/test/feature/voice.feature.test.ts:60-75
Pipeline States
The pipeline manages a finite state machine to inform the UI of the agent's current activity. These states are critical for controlling microphone access and UI indicators.
| State | Description |
|---|---|
| LISTENING | The agent is active and processing audio input for VAD/STT. |
| THINKING | Speech has ended; the LLM is generating a response. |
| SPEAKING | The TTS engine is synthesizing audio or the system is playing a reply. |
| COOLDOWN | Brief state after speaking to prevent the mic from catching the speaker output. |
| ERROR | A component failed during the turn. |
Sources: bindings/electron/src/api/types.ts:108-112, bindings/electron/test/feature/voice.feature.test.ts:245-260, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:41-45
State Transition Sequence
The sequence diagram below shows a typical interaction turn.
Implementation Details
Initialization and Configuration
The pipeline uses a two-phase initialization. Phase 1 registers the native backends and platform adapters. Phase 2 loads the specific models and completes service setup.
- Compose Config: Defines which model IDs to use for STT, LLM, and TTS, and configures VAD thresholds.
- Split-WASM Support (Web): In Web environments, if STT/TTS and LLM live in different DedicatedWorkers, a
CrossWasmVoiceAgentProviderroutes calls between workers. - Native Handle (Mobile/Desktop): On platforms like Android or Electron, a native C++ handle (
rac_voice_agent_t) is used for optimal performance.
Sources: bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:130-150, bindings/kotlin/AGENTS.md:75-90, bindings/react-native/example/AGENTS.md:55-65
Audio Ingress Modes
The pipeline supports three distinct ways to ingest audio data:
- Turn: A single batch of audio is processed to completion.
- Feed: Raw PCM frames are pushed continuously (e.g., from a microphone). The internal energy endpointer or VAD decides when to close the utterance.
- Say: A purely text-to-speech command that bypasses the STT/LLM stages to provide direct feedback.
Sources: bindings/electron/test/feature/voice.feature.test.ts:150-180, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:110-125
Summary
The Voice Agent Pipeline provides a high-level abstraction for complex multi-model AI interactions. By managing state transitions, audio buffering, and component synchronization, it allows developers to implement sophisticated voice interfaces with minimal boilerplate code. Its architecture ensures that whether running on a web browser via WASM or natively on a mobile device, the conversational flow remains consistent and responsive.
Sources: AGENTS.md, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+VoiceAgent.ts:880-900