Speaker Diarization Support
Relevant source files
The following files were used as context for generating this wiki page:
Speaker Diarization in the mlx-qwen3-asr project provides the capability to identify "who spoke when" by attributing segments of transcribed text to specific speaker labels. This feature is implemented as an optional extension that integrates with the pyannote.audio framework to perform speaker-turn inference while maintaining a zero-dependency core ASR pipeline for standard transcription tasks.
The system works by delegating temporal speaker detection to a diarization pipeline and then reconciling those time-based turns with the word-level or chunk-level timestamps generated by the Qwen3-ASR model. This enables the generation of speaker-labeled transcripts in various formats like JSON, Text, or Subtitles.
Sources: mlx_qwen3_asr/diarization.py:1-10, CLAUDE.md:100-110
Architecture and Integration
The diarization module is designed to be "pluggable." While the core ASR pipeline (Audio → Mel → Encoder → Decoder → Text) avoids PyTorch and heavy dependencies, diarization optionally utilizes pyannote.audio (and consequently PyTorch) only when the --diarize flag is enabled.
Component Interaction Flow
The following diagram illustrates how the diarization module interacts with the main ASR process to produce a labeled transcript.
The system performs parallel paths: the ASR pipeline generates the text and timestamps, while the diarization pipeline identifies speaker boundaries. The diarize_word_segments or diarize_chunk_items functions then map the text segments to the most likely speaker based on temporal overlap.
Sources: mlx_qwen3_asr/diarization.py:65-100, mlx_qwen3_asr/cli.py:590-610, CLAUDE.md:120-130
Configuration and Validation
Diarization behavior is controlled via the DiarizationConfig dataclass. Users can specify a fixed number of speakers or provide a range for automatic detection.
Configuration Options
| Parameter | CLI Argument | Type | Default | Description |
|---|---|---|---|---|
num_speakers | --num-speakers | int | None | Fixed number of speakers to detect. |
min_speakers | --min-speakers | int | 1 | Minimum speakers for auto-detection. |
max_speakers | --max-speakers | int | 8 | Maximum speakers for auto-detection. |
Validation is performed by validate_diarization_config to ensure min_speakers is at least 1 and max_speakers is not less than the minimum.
Sources: mlx_qwen3_asr/diarization.py:22-30, mlx_qwen3_asr/cli.py:320-335
Speaker Turn Inference
The inference process delegates the heavy lifting to a pre-trained pyannote model, specifically pyannote/speaker-diarization-community-1 by default.
Backend Initialization
The backend is initialized via _load_pyannote_pipeline(), which handles authentication using environment variables:
PYANNOTE_AUTH_TOKENHF_TOKENHUGGINGFACE_TOKEN
Inference Logic
The function infer_speaker_turns converts the raw audio into a format suitable for the pyannote pipeline. It includes retry logic to handle different versions of the pyannote API, specifically catching TypeError if certain speaker-count arguments are not supported by the loaded model.
Reconciliation and Merging
Once speaker turns (time intervals with labels) and ASR segments (time intervals with text) are available, they must be merged.
Speaker Attribution Logic
The system attributes a speaker to a text segment using the following priority:
- Overlap: The speaker with the maximum temporal overlap with the text segment is chosen.
- Proximity: If no overlap exists (e.g., ASR timestamps and diarization boundaries are slightly desynced), the speaker whose turn midpoint is closest to the text segment midpoint is chosen.
Sources: mlx_qwen3_asr/diarization.py:311-335
Segment Merging
To improve transcript readability, _merge_speaker_segments combines adjacent segments if they belong to the same speaker and the gap between them is below a threshold (default 0.8 seconds for text chunks).
# Example of turn merging logic in diarization.py
def _merge_speaker_turns(turns: list[dict], *, max_gap_sec: float = 0.2) -> list[dict]:
if not turns: return []
merged = []
for t in turns:
if not merged:
merged.append(dict(t))
continue
prev = merged[-1]
if prev["speaker"] == t["speaker"] and float(t["start"]) - float(prev["end"]) <= max_gap_sec:
prev["end"] = max(float(prev["end"]), float(t["end"]))
else:
merged.append(dict(t))
return merged
Sources: mlx_qwen3_asr/diarization.py:293-310, mlx_qwen3_asr/diarization.py:338-360
Evaluation and Benchmarking
The project includes dedicated scripts for evaluating diarization quality using standard metrics: Diarization Error Rate (DER) and Jaccard Error Rate (JER).
Evaluation Workflow
- Manifest Building:
scripts/build_diarization_manifest.pycreates deterministic test sets by mixing single-speaker clips with known ground truth boundaries. - Metric Calculation:
scripts/eval_diarization.pyruns the inference and compares it against the manifest. - Collar Support: Evaluation supports a
collar_sec(default 0.25s) to ignore small desyncs at speaker boundaries during scoring.
Sources: scripts/build_diarization_manifest.py:1-20, scripts/eval_diarization.py:150-180
Comparison Metrics Table
| Metric | Definition | Implementation Detail |
|---|---|---|
| DER | Diarization Error Rate | Sum of Miss, False Alarm, and Confusion time divided by total reference time. |
| JER | Jaccard Error Rate | Evaluates speaker-wise Jaccard similarity to account for speaker counting errors. |
| Confusion | Speaker Mismatch | Calculated after finding the best one-to-one assignment between hypothesis and reference speakers. |
Sources: scripts/eval_diarization.py:225-275
Summary
Speaker Diarization support in mlx-qwen3-asr provides a robust, optional layer for multi-speaker transcription. By delegating speaker detection to pyannote.audio while maintaining internal logic for temporal reconciliation and segment merging, the system produces clean, speaker-labeled transcripts optimized for meeting notes and conversational analysis.
Sources: CLAUDE.md:140-145, mlx_qwen3_asr/diarization.py:1-10