Sherpa-ONNX Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Sherpa-ONNX engine serves as the primary on-device speech processing backend within the project. It provides capabilities for Speech-to-Text (STT), Text-to-Speech (TTS), and Voice Activity Detection (VAD). The engine wraps the upstream Sherpa-ONNX C API and bundles its own ONNX Runtime (ORT) for execution on the CPU. It is categorized as a "Pattern 3" engine, meaning it functions as specialized inference code running on a device runtime.
Sources: engines/AGENTS.md:52-54, engines/AGENTS.md:162-171
The engine exists as a standalone rac_backend_sherpa plugin artifact, distinct from the generic onnx engine. It registers its primitives into the global plugin registry with a priority of 90. The implementation supports multiple platforms, including iOS, Android, macOS, Linux, Windows, and WebAssembly (WASM).
Sources: engines/sherpa/CMakeLists.txt:10-12, engines/AGENTS.md:52-54
Architecture and Integration
The Sherpa-ONNX engine integrates into the core/ commons library through a vtable-based plugin system. It populates specific slots in the rac_engine_vtable_t to declare support for speech modalities.
Supported Modalities
| Modality | vtable Slot | Implementation Description |
|---|---|---|
| STT | stt_ops | Offline recognizer using ONNX models. |
| TTS | tts_ops | Speech synthesis via Piper/ONNX. |
| VAD | vad_ops | Silero-style Voice Activity Detection. |
Sources: engines/AGENTS.md:31-33, engines/AGENTS.md:52-54
Plugin Registration Flow
The engine uses different registration paths depending on whether the host platform requires static or dynamic linking. On static hosts like iOS and WASM, it uses RAC_STATIC_PLUGIN_REGISTER(sherpa). On dynamic hosts like Android and Linux, it builds as a shared library (librac_backend_sherpa.so) loaded via rac_registry_load_plugin().
Model Inspection and Contracts
The engine includes a SherpaOnnxEncoderContract to identify the capabilities of provided ONNX models. This inspector reads the onnx.ModelProto to determine if a model is intended for online (streaming) or offline use.
Encoder Classification Logic
The inspector scans model inputs for specific cache-related strings to differentiate between OnlineTransducer and Offline kinds.
| Encoder Kind | Required Input String |
|---|---|
| OnlineTransducer | cache_last_channel, cache_last_time, cache_last_channel_len |
| Offline | Absence of streaming cache inputs |
| Multilingual | prompt_index |
Build System and Dependencies
The engine relies on prebuilt Sherpa-ONNX artifacts discovered during the CMake configuration phase. These artifacts are platform-specific and must be fetched before compilation.
Dependency Discovery
The CMakeLists.txt file performs per-platform discovery of prebuilt libraries. It looks for static archives (.a) on iOS, macOS, and WASM, and shared libraries (.so or .dll) on Android, Linux, and Windows.
Key dependencies linked with the engine include:
sherpa-onnx-c-api(Public C API)sherpa-onnx-core(Kernels)kaldi-native-fbank-core(Feature extraction)piper_phonemize,espeak-ng,ucd(TTS specific)
Sources: engines/sherpa/CMakeLists.txt:127-219, bindings/web/wasm/scripts/vendor-sherpa-onnx-wasm.sh:42-56
WASM Specifics
For WebAssembly, the engine requires static archives as it cannot link a standalone Emscripten executable. The build process uses emcmake to generate these libraries, ensuring support for threads and exceptions.
emcmake cmake \
-B "${BUILD_DIR}" \
-S "${SRC_DIR}" \
-DSHERPA_ONNX_ENABLE_WASM=ON \
-DSHERPA_ONNX_ENABLE_C_API=ON \
-DBUILD_SHARED_LIBS=OFF
Sources: engines/sherpa/CMakeLists.txt:20-45, bindings/web/wasm/scripts/vendor-sherpa-onnx-wasm.sh:293-315
Platform Availability and Configuration
The engine maintains a SHERPA_ONNX_AVAILABLE flag. If prebuilts are missing, the engine compiles as a non-routable stub that returns RAC_ERROR_BACKEND_UNAVAILABLE during capability checks.
Configuration Options
| Option | Type | Description |
|---|---|---|
RAC_BACKEND_SHERPA | BOOL | Master toggle to build the backend (Default: ON). |
SHERPA_ONNX_ROOT | PATH | Root directory for vendor prebuilts. |
SHERPA_ONNX_VERSION | STRING | Pinned version of the Sherpa-ONNX runtime. |
Sources: engines/sherpa/CMakeLists.txt:46-50, core/scripts/linux/download-sherpa-onnx.sh:26-30
Conclusion
The Sherpa-ONNX engine provides a high-performance, on-device solution for speech processing across the project's supported platforms. By wrapping a specialized C API and utilizing ONNX-based models, it ensures consistent STT, TTS, and VAD behavior. Its architecture allows for flexible deployment as either a static or dynamic plugin while maintaining strict versioning and capability discovery through internal model inspection.
Sources: engines/AGENTS.md:52-54, engines/sherpa/CMakeLists.txt:360-365