Speech-to-Text (STT)

Relevant source files

The following files were used as context for generating this wiki page:

• ios/Runner/NativeSttBridge.swift
• third_party/vad/UPSTREAM_README.md
• ios/Runner/NativeIosTtsBridge.swift
• android/app/src/main/kotlin/app/cogwheel/conduit/NativeTtsBridge.kt
• README.md
• third_party/vad/CHANGELOG.md

Speech-to-Text (STT) in Conduit enables voice-driven interactions by transcribing spoken language into text on the user's device. The system supports live dictation during chats and voice-call modes, integrating with native platform recognizers and third-party Voice Activity Detection (VAD) libraries to optimize performance and battery life.

The architecture relies on bridge components that communicate between the Flutter UI and native platform APIs (iOS SFSpeech and Android RecognitionService). It prioritizes on-device processing to maintain privacy and reduce latency, utilizing libraries like Silero VAD for efficient speech detection.

Sources: README.md:143-145, ios/Runner/NativeSttBridge.swift:1-15

Core Architecture and Lifecycle

The STT system uses a lifecycle state machine to manage transitions between starting, listening, and stopping. This ensures that audio resources are properly acquired and released, especially when switching between different recognition engines or handling interruptions like incoming calls.

Lifecycle Management

The NativeSttLifecycleState class serializes transitions. Every start command awaits the full teardown of the previous session before initializing new audio resources. This prevents race conditions where multiple recognition tasks might attempt to access the microphone simultaneously.

The diagram shows the serialized transition flow ensuring only one STT session owns audio resources at a time.
Sources: ios/Runner/NativeSttBridge.swift:101-165

iOS Implementation

On iOS, the system utilizes two primary recognition engines: the modern SpeechAnalyzer (available on newer iOS versions) and a fallback SFSpeechNativeSttSession.

Recognition Engines

• SpeechAnalyzer: A modern engine that uses AsyncStream for audio input and supports on-device modules. It handles asset installation through AssetInventory if required modules are missing.
• SFSpeechNativeSttSession: A wrapper around SFSpeechRecognizer. It manages a recognition task and an AVAudioEngine tap to stream audio buffers.

Audio Buffer Management

To prevent blocking the real-time audio thread, a NativeSttPCMBufferPool manages a fixed-capacity pool of AVAudioPCMBuffer objects. This allows the system to copy tap data quickly and process it on a serial background worker queue.

Component	Function
NativeSttPCMBufferPool	Manages preallocated buffers for tap callbacks.
NativeSttAudioAppendGate	Synchronously drains audio appends before teardown.
NativeSttResourceMutationGate	Prevents concurrent access to AVAudioEngine.

Sources: ios/Runner/NativeSttBridge.swift:509-560, ios/Runner/NativeSttBridge.swift:806-830, ios/Runner/NativeSttBridge.swift:1367-1380

Voice Activity Detection (VAD)

Conduit integrates a VAD library to detect speech presence, allowing the app to trigger recognition only when the user is speaking. This component uses Silero VAD models (v4 and v5) powered by ONNX Runtime.

VAD Event Flow

The VadHandler emits specific events based on the processed audio frames:

1. onSpeechStart: Initial detection of potential speech.
2. onRealSpeechStart: Confirmed speech that exceeds a minimum frame threshold, filtering out "misfires".
3. onSpeechEnd: Emitted when the user stops speaking, providing the accumulated audio samples.
4. onFrameProcessed: Real-time probabilities and raw samples for each frame.

This sequence illustrates how audio frames are processed through VAD to confirm real speech before triggering transcription logic.
Sources: third_party/vad/UPSTREAM_README.md:275-300, third_party/vad/CHANGELOG.md:110-120

CallKit Integration

The STT system coordinates with CallKit on iOS to handle audio routing during voice calls. When a voice call is active, a NativeResponseWaitCaptureOwner manages a silent audio tap. This keeps the app's audio session active in MODE_IN_COMMUNICATION, ensuring the system does not reclaim the audio route when speech recognition pauses between utterances.

Key Logic for Voice Calls:

• Audio Session Persistence: Uses AVAudioSession.sharedInstance().setActive(true) with notifyOthersOnDeactivation.
• Silent Keep-Alive: Similar to the Android SilentCallAudioKeepAlive implementation, this prevents the OS from dropping the communication route during silence.

Sources: ios/Runner/NativeSttBridge.swift:348-400, android/app/src/main/kotlin/app/cogwheel/conduit/NativeTtsBridge.kt:473-500

API and Configuration

The STT bridge exposes methods to Flutter via MethodChannel and streams events via EventChannel.

Method Channel API

Method	Parameters	Description
checkAvailability	localeId, allowOnlineFallback	Verifies if a recognizer is available for the given locale.
start	localeId, emitAudio, accumulateResults	Starts a recognition session.
stop	N/A	Ends the current session and drains pending audio.
transcribeRecording	pcm, localeId	Transcribes a static audio buffer on-device.

Event Stream Types

• status: Reports engine state (e.g., "downloading", "listening").
• result: Contains transcribed text, marked as final or partial.
• audio: Emits PCM data if emitAudio was requested during start.
• error: Reports failure codes like AUDIO_PRESERVATION_FAILED or SPEECH_ANALYZER_ERROR.

Sources: ios/Runner/NativeSttBridge.swift:650-700, ios/Runner/NativeSttBridge.swift:760-780

Conclusion

The Speech-to-Text (STT) module provides a robust, low-latency interface for voice interaction. By leveraging native lifecycle management, efficient buffer pooling, and advanced VAD models, it ensures reliable transcription while maintaining platform-specific requirements for audio routing and user privacy.