Voice Activity Detection (VAD)
Relevant source files
The following files were used as context for generating this wiki page:
Voice Activity Detection (VAD) identifies human speech within an audio stream. It serves as a critical component for Voice Agents and standalone speech analysis by segmenting audio into "speech" and "silence" periods. This segmentation minimizes unnecessary processing by downstream models like Speech-to-Text (STT) and facilitates natural turn-taking in conversational interfaces.
Sources: idl/vad_options.proto, core/src/features/vad/vad_module.cpp:24-34
Architecture and Components
The VAD system utilizes a dual-backend architecture. It primarily attempts to use a model-based detector (e.g., Silero) if one is loaded. If no model is present, the system automatically falls back to a built-in energy-based detector. This ensure that VAD functionality is always available even without external model dependencies.
The architecture consists of three main layers:
- Service Layer: Dispatches requests to specific backends like Silero ONNX or the internal energy VAD.
- Component Layer: Manages the lifecycle of a VAD instance, including configuration, state tracking, and callback routing.
- Protocol Layer: Handles Protobuf serialization for cross-language SDK communication.
Sources: core/src/features/vad/vad_module.cpp:32-42, core/src/features/vad/rac_vad_service.cpp:154-180
Logic Flow
The following diagram illustrates how the system chooses between model-based detection and the energy-based fallback.
The system prioritizes model accuracy while maintaining an energy-based baseline for low-power or model-less environments.
Sources: core/src/features/vad/vad_module.cpp:520-535, core/src/features/vad/rac_vad_service.cpp:160-175
Detection Mechanisms
Energy-Based Detector
The energy detector calculates the Root Mean Square (RMS) energy of audio frames. It compares this value against a threshold to determine speech activity. It includes features for automatic calibration against ambient noise and logic to prevent feedback loops during Text-to-Speech (TTS) playback by temporarily increasing thresholds.
Key parameters for energy detection:
- Activation Threshold: The normalized sensitivity [0, 1].
- Calibration Multiplier: Scales the ambient noise floor to set the threshold.
- TTS Multiplier: Increases sensitivity during agent speech to avoid self-triggering.
Sources: core/include/rac/features/vad/rac_vad_energy.h:35-70, core/src/features/vad/energy_vad.cpp:200-245
Model-Based Detector
Model detectors use machine learning models (typically ONNX-based) to predict speech probability. These models generally offer higher precision than energy detection, particularly in noisy environments. The rac_vad_service registry manages these plugins.
Sources: core/src/features/vad/rac_vad_service.cpp:154-180, idl/vad_options.proto:90-100
Configuration and Options
VAD behavior is controlled via load-time configurations and per-call options.
Load-Time Configuration (VADConfiguration)
| Field | Type | Default | Description |
|---|---|---|---|
sample_rate | int32 | 16000 | Audio capture rate (8kHz to 48kHz). |
frame_length_ms | int32 | 100 | Length of each analysis window in milliseconds. |
activation_threshold | float | 0.5 | Normalized sensitivity [0, 1]. |
enable_auto_calibration | bool | false | Enables ambient noise floor measurement. |
Sources: idl/vad_options.proto:34-70, core/src/features/vad/vad_module.cpp:445-485
Execution Options (VADOptions)
| Field | Type | Default | Description |
|---|---|---|---|
min_speech_duration_ms | int32 | 250 | Debounce: Discards speech shorter than this (e.g., clicks). |
min_silence_duration_ms | int32 | 500 | Hangover: Silence required to end a speech turn. |
prefix_padding_ms | int32 | 300 | Pre-roll audio kept to preserve the start of speech. |
Sources: idl/vad_options.proto:75-105, core/src/features/vad/vad_module.cpp:780-800
Data Models and Events
VAD Result
The system returns a VADResult for each analyzed chunk of audio.
message VADResult {
bool is_speech = 1;
float probability = 2; // [0.0, 1.0] confidence score
float energy = 3; // RMS energy level
int32 duration_ms = 4; // Length of analyzed frame
int64 timestamp_ms = 5; // Milliseconds since epoch
}
Sources: idl/vad_options.proto:130-150
Activity Events
The system emits SpeechActivityEvent messages when detecting transitions between speech and silence.
Integration and Usage
C API Lifecycle
VAD instances are managed via a handle-based C API.
- Create:
rac_vad_component_create(handle)allocates resources. - Configure:
rac_vad_component_configure(handle, config)sets operating parameters. - Initialize:
rac_vad_component_initialize(handle)prepares the selected backend. - Process:
rac_vad_component_process(handle, samples, count, result)performs detection. - Destroy:
rac_vad_component_destroy(handle)releases all memory.
Sources: core/src/features/vad/vad_module.cpp:328-400, core/tests/test_speech_proto_abi.cpp:455-480
Telemetry and Statistics
The VAD module tracks real-time statistics for debugging and UI visualizations, including current energy levels, ambient noise floor, and speech segment counts. Transitions are published to the internal SDK event system for telemetry tracking of session durations.
Sources: idl/vad_options.proto:160-178, core/src/features/vad/vad_module.cpp:245-280
Summary
Voice Activity Detection in this project provides a robust, always-available mechanism for speech segmentation. By combining high-precision model backends with an automatic energy-based fallback, the system ensures reliable performance across diverse hardware and environmental conditions. The integration of debouncing and hangover logic allows for natural conversational flows and efficient downstream processing.
Sources: core/src/features/vad/vad_module.cpp:18-45, idl/vad_options.proto:1-15