Audio Diarization & Segmentation
Relevant source files
The following files were used as context for generating this wiki page:
Audio Diarization and Semantic Segmentation are specialized on-device AI capabilities provided by the RunAnywhere SDK. Diarization identifies "who spoke when" in an audio stream by partitioning audio into segments associated with specific speaker IDs. Segmentation performs pixel-level classification on image data, labeling every pixel according to a defined set of classes.
Sources: idl/diarization.proto:1-10, core/README.md:20-35, idl/segmentation.proto:1-5
Audio Diarization
Diarization identifies speaker changes and active durations without producing a text transcript. It functions as a standalone capability with a dedicated lifecycle and primitive in the C++ core. The system typically utilizes ONNX-based backends, such as Sortformer, to process audio data.
Sources: idl/diarization.proto:5-9, core/README.md:40-45, bindings/electron/test/feature/embeddings.feature.test.ts:140-150
Diarization Data Flow
The following diagram illustrates the offline diarization process where a batch of audio samples is processed to produce speaker segments.
Diarization Configuration
The DiarizationOptions message defines the requirements for audio processing. The engine requires specific audio formats and provides hints for speaker detection.
| Parameter | Type | Description | Constraints |
|---|---|---|---|
sample_rate | int32 | Input sample rate | Must be 16000 Hz |
channels | int32 | Number of audio channels | Must be 1 (mono) |
encoding | Enum | Audio byte layout | PCM_F32_LE or PCM_S16_LE |
threshold | float | Sensitivity for detection | Range 0.0 to 1.0 (default 0.5) |
max_speakers | int32 | Upper bound hint for speaker count | Minimum 1 |
Sources: idl/diarization.proto:18-42
Streaming Diarization
The system supports real-time diarization through a stream of events. These events provide snapshots of the session hypothesis.
Sources: idl/diarization.proto:60-78
Semantic Segmentation
Segmentation labels every pixel of an image. This is typically used for distinguishing foreground from background or identifying specific objects in a frame. In the RunAnywhere architecture, this capability is served by ONNX backends (e.g., Segformer).
Sources: idl/segmentation.proto:1-10, core/README.md:33, bindings/electron/test/feature/embeddings.feature.test.ts:165-175
Segmentation Architecture
Segmentation utilizes raw RGB pixel data. The SDK provides a class mask and a summary of the identified classes.
Segmentation Output Model
The SegmentationResult contains a 1D array representing the 2D pixel grid and information about the detected categories.
| Field | Type | Description |
|---|---|---|
class_mask | bytes | uint16 array where each value corresponds to a class_id |
classes | list | List of SegmentationClass containing labels and pixel counts |
width | int32 | Width of the processed image |
height | int32 | Height of the processed image |
diagnostic_rgba | bytes | Optional image for visual debugging |
Sources: idl/segmentation.proto:34-45, bindings/python/native/module.cpp:1230-1245
Implementation Details
The C++ core defines service handles for both diarization and segmentation. High-level SDKs (Python, Electron) bind to these services through a standardized lifecycle: create, initialize, process (diarize/segment), and destroy.
Lifecycle Management
- Create: Allocates the service handle with a specific model ID.
- Initialize: Loads the model file from the specified path.
- Process: Executes the inference on audio samples or image pixels.
- Destroy: Frees native resources and unloads the model.
Sources: bindings/python/native/module.cpp:1115-1125, bindings/python/native/module.cpp:1180-1195, bindings/python/tests/fake_core.py:400-435
Code Example: Diarization (Python/C++ Bridge)
// From bindings/python/native/module.cpp
int32_t load_diarization_model(const std::string& model_path, std::optional<std::string> id) {
rac_handle_t h = nullptr;
rac_result_t rc = rac_diarization_create(model_id.c_str(), &h);
rc = rac_diarization_initialize(h, model_path.c_str());
return register_handle(g_diar_handles, h);
}
Sources: bindings/python/native/module.cpp:1114-1128
Conclusion
Audio Diarization and Segmentation provide the RunAnywhere SDK with sophisticated spatial and temporal analysis tools. By leveraging ONNX backends, these modules deliver efficient on-device processing for speaker identification and pixel-level image classification, maintaining data privacy by avoiding cloud-based inference.
Sources: core/README.md:5-15, bindings/electron/test/feature/embeddings.feature.test.ts:140-180