Speaker Diarization
Relevant source files
The following files were used as context for generating this wiki page:
Speaker Diarization in MacParakeet is the process of partitioning an audio stream into homogeneous segments according to the speaker identity, answering the question "who spoke when." The system utilizes an offline, on-device pipeline provided by the FluidAudio SDK, primarily for file and YouTube transcriptions, as well as refining the isolated system track in meeting recordings.
The feature is designed to be non-fatal; if diarization fails due to errors like noSpeechDetected, the ASR (Automated Speech Recognition) transcription is still persisted, ensuring that users never lose a transcript due to speaker identification issues. Diarization is controlled via a global "Speaker detection" toggle in Settings, which is off by default to balance accuracy and processing speed.
Sources: spec/adr/010-speaker-diarization.md:6-10, spec/06-stt-engine.md:434-436, plans/completed/2026-03-speaker-diarization.md:12, plans/active/2026-05-speaker-diarization-quality.md:15-18
System Architecture
The diarization architecture follows a modular approach where the ASR engine and the Diarization pipeline run independently, typically sequentially, on the same audio source. The results are later merged based on timestamp alignment.
Processing Pipeline
The system uses the FluidAudio OfflineDiarizerManager which implements a three-stage pipeline:
- Segmentation (Pyannote community-1): Frame-level detection of speech/silence and speaker changes using a powerset segmentation model.
- Embedding Extraction (WeSpeaker v2): Generation of 256-dimensional voice fingerprints for speech segments.
- Clustering (VBx + AHC): Grouping of fingerprints by voice similarity to assign stable, consistent speaker IDs (e.g., "S1", "S2").
Sources: spec/adr/010-speaker-diarization.md:41-52, spec/06-stt-engine.md:440-452
The following diagram illustrates the high-level data flow for a transcription with diarization enabled:
Components and Data Models
DiarizationService
The DiarizationService wraps the FluidAudio OfflineDiarizerManager. It is responsible for initializing models, normalizing speaker IDs (e.g., converting FluidAudio's speaker_0 to S1), and executing the diarization process.
Key structures include:
DiarizationOptions: Allows passing speaker count hints (exact, minimum, or maximum) to the clustering algorithm.SpeakerSegment: Represents a time-bounded segment (startMstoendMs) assigned to a specificspeakerId.
Sources: plans/completed/2026-03-speaker-diarization.md:71-105, plans/active/2026-05-speaker-diarization-quality.md:70-85
Speaker Merger (Logic)
The merging logic is a pure function that iterates through word timestamps and speaker segments. For each word, it identifies the diarization segment with the maximum time overlap and assigns that segment's speakerId to the word.
- Overlap Rule: If a word overlaps multiple segments, the one with the most overlap wins.
- Tie-breaking: If overlap is equal, the earlier segment is chosen.
- Unassigned Words: Words falling in silence gaps or overlapping speech regions where the diarizer trims output are assigned
nilforspeakerId.
Sources: plans/completed/2026-03-speaker-diarization.md:121-137, plans/active/2026-05-speaker-diarization-quality.md:195-210
Data Model Storage
Diarization data is persisted in the Transcription model using JSON-formatted columns:
| Field | Description | Type |
|---|---|---|
WordTimestamp.speakerId | Stable raw ID from the pipeline (e.g., "S1"). | String? |
Transcription.speakers | Mapping of stable IDs to display labels (e.g., "Speaker 1" or "Sarah"). | [SpeakerInfo]? |
Transcription.diarizationSegments | Raw diarization output for analytics and UI features. | [DiarizationSegmentRecord]? |
Transcription.speakerCount | Total number of distinct speakers detected. | Int? |
Sources: plans/completed/2026-03-speaker-diarization.md:33-58, spec/adr/010-speaker-diarization.md:92-97
Model Specifications
The pipeline requires specific CoreML models which are downloaded during onboarding if speaker detection is enabled.
| Component | Model | Size | Runtime |
|---|---|---|---|
| Segmentation | Pyannote community-1 (powerset) | ~50 MB | Apple Neural Engine (ANE) |
| Embeddings | WeSpeaker v2 (256-dim) | ~40 MB | ANE |
| Filter bank | Fbank feature extractor | ~1 MB | CPU |
| PLDA Scoring | PLDA rho model + psi parameters | ~10 MB | CPU |
| Total | ~130 MB |
Sources: spec/06-stt-engine.md:454-463, spec/adr/010-speaker-diarization.md:71-82
Quality and Constraints
Speaker Count Hints
Users can provide constraints to the diarizer to improve clustering accuracy. These are mapped to OfflineDiarizerConfig.Clustering parameters:
numSpeakers: Used when the exact number of speakers is known.minSpeakers/maxSpeakers: Defines a range for the clustering algorithm to operate within.
Sources: plans/active/2026-05-speaker-diarization-quality.md:27-35, 110-125
Performance Metrics
Diarization is optimized for speed on Apple Silicon, typically processing an hour of audio in approximately 30-60 seconds.
| Metric | Estimated Value |
|---|---|
| Diarization Speed | 64-122x Real-time (RTF) |
| Combined (ASR + Diarization) | ~53-79 seconds per hour of audio |
| Accuracy (DER) | ~15% on VoxConverse, ~17.7% on AMI |
Sources: spec/06-stt-engine.md:493-501, spec/adr/010-speaker-diarization.md:139-142
User Interface and Interaction
The UI manages the relationship between stable raw IDs (provided by the engine) and user-facing labels.
UI Features
- Speaker turns: Consecutive words by the same
speakerIdare grouped into turns with colored headers. - Analytics: Per-speaker statistics including speaking time (summed from
diarizationSegments) and word counts are displayed in a summary panel. - Export: Speaker labels are included in TXT, Markdown, SRT, VTT, DOCX, and PDF formats. SRT/VTT cues are split specifically at speaker boundaries to maintain attribution accuracy.
Sources: plans/completed/2026-03-speaker-diarization.md:173-181, 197-206, 212-220
Diarization significantly enhances the utility of long-form transcriptions by providing structural context. By utilizing stable IDs and separate display mappings, the system allows for flexible user corrections without requiring expensive re-processing of word timestamps.
Sources: plans/completed/2026-03-speaker-diarization.md:277-285, spec/adr/010-speaker-diarization.md:120-126