Benchmark Manifest Builders
Relevant source files
The following files were used as context for generating this wiki page:
Benchmark Manifest Builders are a suite of utility scripts designed to create deterministic dataset manifests for evaluating Automatic Speech Recognition (ASR) performance. These builders transform raw data from sources like HuggingFace datasets or existing JSONL manifests into structured, reproducible evaluation sets targeting specific ASR challenges, including real-world conversational speech, long-form audio, and multi-speaker diarization. Sources: scripts/README.md
The primary output of these builders is a .jsonl manifest file accompanied by a directory of standardized 16kHz WAV audio files. These artifacts are consumed by evaluation tools like scripts/eval_manifest_quality.py to measure Word Error Rate (WER) and Character Error Rate (CER), or scripts/eval_streaming_manifest.py for real-time stability diagnostics. Sources: scripts/eval_manifest_quality.py:1-20, scripts/eval_streaming_manifest.py:1-15
Core Architecture and Data Flow
The manifest building process generally follows a three-stage pipeline: ingestion of source data (local or remote), deterministic sampling or transformation, and artifact generation (audio writing and manifest serialization).
The diagram shows the transition from raw data sources to standardized evaluation artifacts via deterministic processing logic.
Sources: scripts/build_realworld_manifest.py:20-50, scripts/build_diarization_manifest.py:80-120
Real-World and Long-Form Manifest Construction
Builders for real-world data extract samples from established datasets like ami_ihm_test (meeting speech) and earnings22 (financial calls). These scripts prioritize speaker diversity and specific duration constraints to ensure the benchmark reflects conversational complexity. Sources: scripts/build_realworld_manifest.py:10-30
Real-World Data Ingestion
The build_realworld_manifest.py script utilizes SourceSpec dataclasses to map HuggingFace parquet fields to internal structures. It employs a round-robin speaker sampling strategy to ensure balanced representation across the benchmark. Sources: scripts/build_realworld_manifest.py:24-42, scripts/build_realworld_manifest.py:155-185
| Component | Description | Relevant File |
|---|---|---|
SourceSpec | Defines repo ID, parquet prefixes, and field mappings for HF datasets. | build_realworld_manifest.py |
_load_candidates | Iteratively downloads HF shards and filters samples by duration and content. | build_realworld_manifest.py |
_sample_speaker_round_robin | Ensures deterministic selection of unique speakers based on a provided seed. | build_realworld_manifest.py |
Long-Form Manifest Generation
Long-form benchmarks are created via two methods:
- Synthetic Concatenation:
build_longform_manifest.pytakes existing short-form manifest rows and concatenates them with configurable silence segments (default 0.25s) to reach a target duration (default 75s). Sources: scripts/build_longform_manifest.py:75-105 - Natural Extraction:
build_earnings22_longform_manifest.pyextracts full financial call segments fromdistil-whisper/earnings22, typically ranging from 850 to 1500 seconds. Sources: scripts/build_earnings22_longform_manifest.py:130-150
Multi-Speaker Diarization Builders
The build_diarization_manifest.py script creates complex audio scenarios by mixing single-speaker clips into sequential multi-speaker tracks. This is critical for testing speaker-labeled transcription and diarization accuracy. Sources: scripts/build_diarization_manifest.py:1-10
The sequence illustrates the transformation of isolated speaker recordings into mixed diarization benchmarks with precise segment metadata.
Sources: scripts/build_diarization_manifest.py:110-170, scripts/build_diarization_manifest.py:180-210
Manifest Schema and Field Definitions
The manifest files serve as the ground truth for downstream evaluation. Each row represents a single audio sample and its associated metadata. Sources: scripts/eval_manifest_quality.py:35-50
| Field | Type | Description |
|---|---|---|
sample_id | string | Unique identifier for the evaluation sample. |
audio_path | string | Absolute or relative path to the standardized 16kHz WAV file. |
language | string | The ground-truth language of the audio. |
reference_text | string | The ground-truth transcription (raw or normalized). |
duration_sec | float | The length of the audio in seconds. |
speaker_id | string | Identifier for the speaker(s). Often "mixed" for diarization or long-form. |
reference_speaker_segments | list[dict] | (Diarization only) Start/end times for each speaker in a mix. |
Sources: scripts/build_diarization_manifest.py:200-215, scripts/build_realworld_manifest.py:270-285, scripts/eval_manifest_quality.py:100-120
Implementation Details
Audio Standardization
All builders utilize a shared utility for audio processing, ensuring that all benchmark audio is converted to a 16kHz mono float32 format before writing to disk. This is typically handled by mlx_qwen3_asr.audio.load_audio combined with soundfile. Sources: scripts/build_realworld_manifest.py:202-212, scripts/build_longform_manifest.py:53-61
def _read_audio_16k(path: Path) -> np.ndarray:
import soundfile as sf
from mlx_qwen3_asr.audio import load_audio
audio, sr = sf.read(str(path), dtype="float32")
if audio.ndim > 1:
audio = np.mean(audio, axis=1).astype(np.float32)
return np.array(load_audio((audio, int(sr)))).astype(np.float32)
Sources: scripts/build_longform_manifest.py:53-61
Deterministic Sampling
To ensure reproducibility across different environments, builders use numpy.random.default_rng(seed) for all sampling decisions, including speaker selection and mix planning. This allows developers to recreate the exact same benchmark artifacts using the same input data and seed value. Sources: scripts/build_realworld_manifest.py:165-175, scripts/build_diarization_manifest.py:80-90
Benchmark Manifest Builders provide the foundation for a rigorous, data-driven evaluation pipeline. By automating the creation of targeted manifests, the project ensures that performance gains in the MLX implementation are validated against diverse and challenging audio conditions. Sources: scripts/README.md, scripts/quality_gate.py:400-450