Core Correctness Invariants
Relevant source files
The following files were used as context for generating this wiki page:
Core Correctness Invariants represent the non-negotiable architectural and implementation requirements for the mlx-qwen3-asr package. These invariants ensure that the MLX reimplementation of Qwen3-ASR maintains parity with the original HuggingFace weights and reference PyTorch implementation. Violating these invariants typically results in silent inference failures or degraded output quality.
The project prioritizes a "zero-PyTorch" core path, meaning the primary ASR pipeline (audio to text) must not import torch. Correctness is enforced through a rigorous quality gate involving reference parity suites, LibriSpeech benchmarks, and manifest quality evaluations.
Sources: CLAUDE.md:1-25, CLAUDE.md:65-72
Architectural Invariants
The architecture follows a strict encoder-decoder structure where specific normalization and embedding rules apply to different components.
MRoPE and Positional Embeddings
The text decoder utilizes interleaved Multi-RoPE (MRoPE). Unlike standard chunked implementations, MRoPE in Qwen3-ASR is interleaved with specific sections [24, 20, 20] and a stride-3 frequency assignment across temporal, height, and width dimensions. Additionally, sinusoidal position embeddings are computed dynamically during runtime rather than being loaded from weight files.
Sources: CLAUDE.md:73-77, CLAUDE.md:80-81
Normalization and Bias
Consistency in normalization layers is critical for model stability. The audio encoder utilizes LayerNorm with bias, whereas the text decoder uses RMSNorm without bias. Mixing these configurations breaks inference.
Sources: CLAUDE.md:78-79
Weight Transposition
During model conversion from PyTorch to MLX, Conv2d weights must be transposed from the PyTorch format (out, in, kH, kW) to the MLX format (out, kH, kW, in) using transpose(0, 2, 3, 1).
Sources: CLAUDE.md:80-81
Token and Language Invariants
The system relies on specific Reserved Token IDs and language validation to ensure the decoder behaves predictably.
| Token Type | ID | Description |
|---|---|---|
| Audio Start | 151669 | Signals the beginning of audio features |
| Audio End | 151670 | Signals the end of audio features |
| Audio Pad | 151676 | Placeholder for audio feature injection |
| EOS Tokens | {151643, 151645} | End-of-sentence markers used for trimming |
Sources: CLAUDE.md:82-84, scripts/eval_nonnear_parity_matrix.py:18-18, scripts/eval_reference_parity_suite.py:34-34
Special Token Handling
Correctness requires aggressive trimming of End-of-Sentence (EOS) tokens from the generated sequence to prevent trailing noise in parity comparisons.
def _trim_eos(tokens: list[int]) -> list[int]:
out = list(tokens)
while out and out[-1] in EOS_IDS:
out.pop()
return out
Sources: scripts/eval_reference_parity_suite.py:168-173, scripts/eval_nonnear_parity_matrix.py:84-89
Parity and Evaluation Invariants
The project maintains correctness through a "Reference Parity Suite" that compares MLX outputs against the official qwen-asr implementation.
Mismatch Categorization
Mismatches between MLX and the reference are classified into specific categories to identify regression types:
- exact_match: Token-level identity.
- punctuation_or_tokenization: Differences only in non-alphanumeric characters.
- numeric_surface_form: Differences in how numbers are represented (e.g., "53" vs "fifty-three").
- minor_lexical_shift: Small edit distance ratio (<= 0.12).
- content_shift: Significant deviation in transcription content.
Sources: scripts/eval/analyze_reference_parity_mismatches.py:32-51
Data Flow for Parity Validation
The following diagram illustrates the flow used to validate MLX output against the PyTorch reference during the quality gate.
The flow ensures that audio is processed by both systems simultaneously, with identical pre-processing and post-generation token trimming.
Sources: scripts/eval_reference_parity_suite.py:365-420, scripts/eval/analyze_reference_parity_mismatches.py:32-51
Streaming Correctness
In streaming mode, specific metrics are used to ensure the stability of partial transcriptions.
| Metric | Definition |
|---|---|
partial_stability | The consistency of partial results before finalization. |
rewrite_rate | The frequency at which the system updates previously "stable" text. |
finalization_delta_chars | The character difference between the last partial and the final result. |
Sources: CLAUDE.md:43-45, scripts/eval_streaming_manifest.py:113-116
Streaming Logic Flow
Streaming involves maintaining a Session state and managing KV-cache incremental decoding.
The streaming process uses CJK-aware splitting to manage stable and unstable tokens, particularly for languages like Chinese, Japanese, and Korean where character boundaries are critical.
Sources: CLAUDE.md:40-42, scripts/eval_streaming_manifest.py:228-245
Summary of Metric Invariants
Correctness is quantified using standard ASR metrics and project-specific parity rates.
- WER (Word Error Rate): Calculated after text normalization which removes non-alphanumeric characters except apostrophes.
- CER (Character Error Rate): Used primarily for CJK languages or when spaces are absent.
- Token Match Rate: The percentage of samples where MLX tokens exactly match reference tokens after EOS trimming.
- Text Match Rate: The percentage of samples where normalized MLX text matches normalized reference text.
Sources: mlx_qwen3_asr/eval_metrics.py:40-62, scripts/eval_reference_parity_suite.py:178-193, scripts/eval_reference_parity_suite.py:448-455