Reference Parity Analysis
Relevant source files
The following files were used as context for generating this wiki page:
Reference Parity Analysis is a critical validation process within the mlx-qwen3-asr project designed to ensure that the MLX implementation produces outputs identical or near-identical to the official PyTorch qwen-asr reference model. This process is essential because the project is a ground-up reimplementation where every layer has been rewritten for Apple Silicon's Metal API, making bit-for-bit or token-for-token validation a non-negotiable correctness invariant.
The analysis scope includes token-level parity, text-level parity (normalized), and timing parity for the forced aligner. It utilizes a variety of datasets, including LibriSpeech and multilingual manifests, to benchmark performance across 52 supported languages. This analysis is integrated into the project's quality gates, ensuring that optimizations or architectural changes do not introduce silent regressions in output quality.
Sources: CLAUDE.md:12-25, scripts/quality_gate.py:596-680, scripts/eval_reference_parity_suite.py:1-13
Architecture and Data Flow
The Reference Parity Analysis system operates by running parallel inference pipelines using the MLX-based model and the PyTorch-based reference model on identical audio inputs.
Parity Execution Flow
The following diagram illustrates the sequence of operations during a parity check:
The system extracts features, generates tokens, and performs dual-layer comparison: strict token matching and normalized text matching.
Sources: scripts/eval_reference_parity_suite.py:347-418, tests/test_reference_parity.py:46-88
Key Data Structures
| Structure | File | Description |
|---|---|---|
LibriSample | scripts/eval_reference_parity_suite.py | Metadata for a LibriSpeech sample (ID, speaker, audio path, subset). |
SuiteSample | scripts/eval_reference_parity_suite.py | Extended sample container including language hints, synthetic audio payloads, and condition labels. |
PreparedSample | scripts/eval_nonnear_parity_matrix.py | A sample with pre-extracted PyTorch features to ensure MLX is tested against identical input tensors. |
CaseConfig | scripts/eval_nonnear_parity_matrix.py | Configuration for a specific probe case (e.g., forcing dense vs. segmented windows). |
Sources: scripts/eval_reference_parity_suite.py:37-56, scripts/eval_nonnear_parity_matrix.py:43-60
Parity Metric Categories
Mismatches between the MLX implementation and the reference are not treated as monolithic failures. The analyze_reference_parity_mismatches.py script categorizes differences to help identify specific implementation gaps.
Mismatch Taxonomy
- exact_match: Tokens are identical.
- punctuation_or_tokenization: The normalized text matches, but specific tokens (like punctuation marks) differ.
- numeric_surface_form: Differences in how numbers are represented (e.g., "53" vs "fifty-three").
- minor_lexical_shift: Edit distance ratio is $\le 0.12$.
- content_shift: Significant deviation in transcript content.
Sources: scripts/eval/analyze_reference_parity_mismatches.py:38-55
Normalization Logic
Normalization is crucial for "text match" validation, especially for language-agnostic comparison. The _normalize_parity_text function performs the following:
- NFKC Normalization: Standardizes Unicode characters.
- Casefolding: Lowercases text.
- Noise Removal: Removes symbols and punctuation while retaining letters, numbers, and marks across all scripts (CJK, Latin, etc.).
- Whitespace Compression: Reduces multiple spaces to a single space and trims ends.
Sources: scripts/eval_reference_parity_suite.py:171-190, scripts/eval/analyze_reference_parity_mismatches.py:39-42
Parity Evaluation Suites
The project provides multiple specialized scripts to probe different aspects of model behavior.
Aligner Parity
The forced aligner (0.6B model) is validated by comparing its word-level timestamps against the qwen_asr backend. It measures the Mean Absolute Error (MAE) in milliseconds for both start_time and end_time of matched words.
Sources: scripts/eval_aligner_parity.py:165-207
Non-near Parity Matrix
This specialized probe identifies "non-near" mismatches—cases where the MLX output deviates significantly. It tests various architectural toggles to find the root cause of divergence:
- Dtype Toggling: Comparing
float16,float32, andbfloat16. - Encoder Window Modes: Forcing dense windows vs. segmented windows (
enc._WINDOWED_SEGMENT_MIN_WINDOWS). - Attention Fallback: Forcing fallback SDPA paths to ensure
mx.fast.scaled_dot_product_attentionis not the source of error.
Sources: scripts/eval_nonnear_parity_matrix.py:127-160, scripts/eval_nonnear_parity_matrix.py:220-226
Reference Parity Suite Configuration
The eval_reference_parity_suite.py script supports extensive command-line configuration for broad validation:
| Parameter | Default | Description |
|---|---|---|
--subsets | test-clean,test-other | LibriSpeech subsets to evaluate. |
--sampling | speaker_round_robin | Strategy to select samples to ensure speaker diversity. |
--include-long-mixes | False | Generates synthetic long clips by concatenating segments. |
--include-noise-variants | False | Injects white noise at specific SNR levels (e.g., 10dB, 5dB). |
--fail-match-rate-below | None | Threshold to trigger a non-zero exit code for CI/CD gates. |
Sources: scripts/eval_reference_parity_suite.py:245-285
Integration with Quality Gates
Parity analysis is a mandatory component of the "Release" quality gate. The quality_gate.py script enforces specific thresholds before a version is considered production-ready.
In strict release mode, the suite requires a minimum token match rate (e.g., 0.58) and text match rate (e.g., 0.61) across multilingual manifests like FLEURS.
Sources: scripts/quality_gate.py:633-680
Conclusion
Reference Parity Analysis serves as the ground-truth validation mechanism for mlx-qwen3-asr. By utilizing a multi-tiered approach—comparing raw tokens, normalized text, and alignment timings—the project maintains high fidelity to the original Qwen3-ASR model while leveraging the performance benefits of MLX and Metal. The categorized mismatch analysis further allows for targeted refinement of the implementation, ensuring that lexical and content shifts are minimized.