Voice Activity Detection (VAD)
Relevant source files
The following files were used as context for generating this wiki page:
Voice Activity Detection (VAD) determines the presence or absence of human speech in an audio stream. Conduit integrates a patched version of the vad Flutter library, which utilizes Silero VAD models (v4 and v5) powered by ONNX Runtime for high-performance cross-platform inference.
The VAD system allows the application to start and stop listening sessions, process audio frames in real-time, and trigger events based on speech start, speech end, or misfires. It serves as a critical component for voice-call modes and real-time speech-to-text (STT) features within the mobile client.
Sources: third_party/vad/UPSTREAM_README.md, third_party/vad/README.md
Architecture and Implementation
Conduit uses a local, patched copy of the vad package (version 0.0.8). This local version includes specific fixes for memory leaks and context handling for Silero VAD v5 to ensure quiet speech is not incorrectly identified as silence.
Component Structure
The system relies on three main layers for processing:
- VadHandler: The high-level API used by the Flutter application to manage listening sessions and event streams.
- VadIterator: Manages the frame-by-frame processing and state transitions of the VAD logic.
- VadInference: Provides platform-specific inference using FFI bindings to ONNX Runtime on native platforms and
dart:js_interopon Web.
The diagram shows the hierarchical flow from the Flutter application down to the hardware-accelerated inference models.
Sources: third_party/vad/CHANGELOG.md, third_party/vad/UPSTREAM_README.md, third_party/vad/README.md
Patch Improvements
Conduit's specific patches modify the standard VAD behavior:
- V5 Context Handling:
SileroV5Model.processnow prepends the last 64 samples of the previous frame to the current 512-sample frame. This prevents scores from falling during quiet speech. - Memory Management: Output tensors are explicitly released after copying values, and name strings are freed in
finallyblocks to reduce leaks. - Typed Audio Conversion: Speech-end and chunk events convert audio directly into typed lists (
Int16List,Float64List) to avoid the overhead of boxed lists.
Sources: third_party/vad/README.md:10-31
API and Configuration
The VadHandler provides the primary interface for configuring and interacting with the VAD system.
Listening Parameters
The startListening method accepts various parameters to tune speech detection sensitivity and buffering.
| Parameter | Type | Default | Description |
|---|---|---|---|
positiveSpeechThreshold | double | 0.5 | Probability threshold to start speech detection. |
negativeSpeechThreshold | double | 0.35 | Probability threshold to end speech detection. |
frameSamples | int | 1536 | Number of samples per frame (512 for v5, 1536 for v4). |
model | String | 'v4' | The VAD model version to use ('v4' or 'v5'). |
minSpeechFrames | int | 3 | Minimum frames required to confirm real speech. |
redemptionFrames | int | 8 | Frames to wait after speech probability drops before ending. |
Sources: third_party/vad/UPSTREAM_README.md, third_party/vad/CHANGELOG.md
Event Streams
Applications can listen to several event streams to react to voice activity:
- onSpeechStart: Triggered when initial speech activity is detected.
- onRealSpeechStart: Emitted when speech is confirmed (exceeds
minSpeechFrames). - onSpeechEnd: Emitted when speech stops, providing the captured audio samples.
- onVADMisfire: Triggered if detected speech did not meet the minimum duration.
- onFrameProcessed: Provides raw audio data and speech probabilities for every processed frame.
Sources: third_party/vad/UPSTREAM_README.md, third_party/vad/CHANGELOG.md
Native Integration
Conduit uses native bridges to manage audio capture and route VAD events to the speech-to-text engines.
iOS Audio Pipeline
On iOS, RealtimeAudioBridge captures microphone input at 24 kHz. It uses AVAudioEngine with voice processing enabled to facilitate echo cancellation. The captured buffers are converted and delivered as PCM frames to the Flutter side for VAD processing.
The sequence diagram illustrates how audio travels from the hardware through the native bridge to the VAD inference engine in Flutter.
Sources: ios/Runner/RealtimeAudioBridge.swift:233-287, ios/Runner/NativeSttBridge.swift:924-950
Android and Echo Cancellation
On Android, the system requests permissions for RECORD_AUDIO and MODIFY_AUDIO_SETTINGS. To resolve echo cancellation issues on specific devices (e.g., Samsung), the system utilizes AudioManager.MODE_IN_COMMUNICATION and speakerphone mode.
Sources: third_party/vad/UPSTREAM_README.md
Implementation Details
Frame Processing Logic
The VAD process operates on a fixed sample rate of 16kHz. When the input audio rate differs, it is automatically upsampled to 16kHz before inference.
// Example of starting a VAD session with Silero V5
await vadHandler.startListening(
model: 'v5',
frameSamples: 512, // Required for v5
positiveSpeechThreshold: 0.6,
negativeSpeechThreshold: 0.4,
);
Sources: third_party/vad/UPSTREAM_README.md, third_party/vad/CHANGELOG.md
Diagnostics and Logging
The system includes a production logging callback (onLog) to route internal VAD state transitions and ONNX Runtime errors to telemetry services like Crashlytics or Sentry. A "breadcrumb" log is emitted immediately before each ONNX Runtime session creation to help diagnose process crashes.
Sources: third_party/vad/CHANGELOG.md, third_party/vad/UPSTREAM_README.md
Summary
Voice Activity Detection in Conduit provides robust, low-latency speech detection across platforms. By leveraging Silero models via ONNX Runtime and applying custom patches for audio context and memory safety, the system ensures reliable performance for conversational AI interactions and dictation features.