Speech-to-Text (STT) & Transcription
Relevant source files
The following files were used as context for generating this wiki page:
Speech-to-Text (STT) and Transcription services within the RunAnywhere SDK provide on-device and hybrid capabilities for converting audio input into text. The system supports batch processing of audio files and real-time streaming transcription, utilizing various backends such as Sherpa-ONNX, QHexRT, and Cloud-based HTTP services.
Sources: core/README.md, idl/stt_options.proto
Architecture and Engine Integration
The STT system operates through a unified C ABI (rac_*) that dispatches requests to specific engine plugins via a vtable. The TRANSCRIBE primitive occupies a specific slot in the rac_engine_vtable_t structure, allowing different backends to handle transcription based on hardware availability and configuration.
Supported Backends
| Backend | Implementation Detail | Target Platform |
|---|---|---|
| Sherpa-ONNX | Offline ONNX-based transcription | Cross-platform |
| QHexRT | Qualcomm Hexagon NPU acceleration | Android (Snapdragon) |
| Cloud STT | Online transcription via HTTP | Cross-platform |
| MLX | Apple MLX-based speech processing | iOS / macOS |
Sources: core/README.md, AGENTS.md
Transcription Data Flow
The following diagram illustrates how an audio source is processed through the SDK to produce transcription results.
The Hybrid Router selects the appropriate engine based on the STTConfiguration and available hardware.
Sources: core/README.md, idl/stt_options.proto
Data Models and Configuration
Transcription behavior is governed by STTConfiguration for initialization and STTOptions for per-call overrides.
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
model_id | string | - | Unique identifier for the transcription model. |
sample_rate | int32 | 16000 | Sampling rate (Min: 8000, Max: 48000). |
enable_vad | bool | false | Enables Voice Activity Detection. |
enable_punctuation | bool | true | Automatically adds punctuation to output. |
enable_word_timestamps | bool | true | Provides start/end times for each word. |
language | string | "" | BCP-47 tag; empty string triggers auto-detect. |
Sources: idl/stt_options.proto:26-45
Audio Input Structures
The STTAudioSource message defines the input format, supporting either raw in-process bytes or a file URI.
message STTAudioSource {
oneof source {
bytes audio_data = 1;
string file_uri = 2;
}
AudioEncoding encoding = 4;
AudioFormat audio_format = 5;
int32 sample_rate = 6;
int32 channels = 7;
int64 duration_ms = 9;
}
Sources: idl/stt_options.proto:69-89
Streaming and Events
The SDK supports streaming transcription where partial results are emitted as the audio is processed. This is managed via an event-based system that uses a sequence-numbered envelope.
STT Stream Event Kinds
STT_STREAM_EVENT_KIND_STARTED: Transcription session initialized.STT_STREAM_EVENT_KIND_PARTIAL: In-progress hypothesis;textreplaces the previous partial.STT_STREAM_EVENT_KIND_FINAL: Finalized text for a segment.STT_STREAM_EVENT_KIND_ENDPOINT: Emitted aftersilence_duration_msis reached.STT_STREAM_EVENT_KIND_ERROR: Transcription failure.
Sources: idl/stt_options.proto:145-167
Sequence Diagram: Streaming Transcription
Implementation Details
Electron/Node.js Usage
In the Electron SDK, transcription is handled by the NativeAddon. Models are loaded explicitly before use.
const sh = await ra.loadSttModel(sttDir, 'sherpa-onnx-whisper-tiny.en');
const transcription = await ra.transcribe(sh, pcm);
await ra.unloadSttModel(sh);
Sources: bindings/electron/native/test_speech.ts:46-51
React Native Integration
The React Native SDK uses AudioCaptureManager for recording and RunAnywhere.stt.transcribe() for processing. It specifically handles Hermes engine limitations by using manual iterators for async streams.
Sources: bindings/react-native/example/README.md, bindings/react-native/example/CLAUDE.md
Kotlin/Android Integration
The Kotlin SDK provides extension functions under public/extensions/ that bridge to the C++ rac_stt_* functions via JNI. It supports coroutines/Flow integration for streaming events.
Sources: bindings/kotlin/CLAUDE.md
Summary
The Speech-to-Text system provides a flexible, multi-backend architecture for transcription. By supporting both offline on-device processing via ONNX and NPU backends and cloud-based hybrid routing, it ensures high-performance transcription across mobile, desktop, and web platforms. The system relies on a strictly defined Protobuf IDL to maintain type-safety and consistency across all SDK bindings.