Evaluation Tooling & Metrics
Relevant source files
The following files were used as context for generating this wiki page:
The mlx-qwen3-asr project implements a multi-layered evaluation framework designed to ensure transcription quality, architectural parity with reference models, and performance stability. The system spans from low-level token edit distance metrics to high-level automated quality gates that block regressions during the release process.
This tooling is critical for validating the MLX implementation against the original PyTorch-based Qwen3-ASR. It includes specialized scripts for batch processing LibriSpeech datasets, analyzing manifest-based quality, and measuring streaming-specific stability metrics such as rewrite rates and partial stability.
Sources: CLAUDE.md, scripts/README.md
Core Quality Metrics
The project utilizes standard Automatic Speech Recognition (ASR) metrics to quantify accuracy. These are primarily based on the Levenshtein distance between reference and hypothesis token sequences.
Word and Character Error Rates
Accuracy is measured via Word Error Rate (WER) and Character Error Rate (CER). The system first normalizes text to lowercase and removes non-alphanumeric characters (excluding apostrophes) to ensure stable comparisons.
| Metric | Calculation Logic | Use Case |
|---|---|---|
| WER | edit_distance(ref_tokens, hyp_tokens) / len(ref_tokens) | Primary metric for Western/Space-delimited languages. |
| CER | edit_distance(ref_chars, hyp_chars) / len(ref_chars) | Primary metric for character-based languages (Chinese, Japanese, Korean). |
Sources: mlx_qwen3_asr/eval_metrics.py:20-56, scripts/eval_manifest_quality.py:65-80
Multi-lingual Metric Selection
The evaluation tooling dynamically selects the "Primary Error Rate" based on the detected or specified language. Languages such as Chinese (zh), Japanese (ja), and Korean (ko) default to CER, while others use WER.
Sources: scripts/eval_manifest_quality.py:83-104
Reference Parity Analysis
A key component of the evaluation suite is ensuring the MLX implementation produces results identical (or "near-identical") to the PyTorch reference implementation.
Mismatch Categorization
When parity is not achieved, the analyze_reference_parity_mismatches.py script categorizes differences to help identify if the delta is a result of minor tokenization artifacts or actual content shifts.
This diagram illustrates the logic for bucketing parity failures into actionable categories.
Sources: scripts/eval/analyze_reference_parity_mismatches.py:32-52
Streaming Diagnostics
For real-time transcription, the system evaluates the stability of the KV-cache and the likelihood of the model "rewriting" its previous output as more audio context becomes available.
Stability Metrics
Streaming evaluation tracks three specific dimensions:
- Partial Stability: The ratio of tokens that remain unchanged between incremental updates.
- Rewrite Rate: The frequency at which the system retracts and replaces previously outputted text.
- Finalization Delta: The difference in characters between the final stable output and the last "unstable" partial.
Sources: scripts/eval_streaming_manifest.py:113-145, CLAUDE.md
Automated Quality Gates
The project implements a centralized quality_gate.py script that orchestrates various evaluation lanes. This gate can be run in fast mode (linting and unit tests) or release mode (full quality, parity, and performance benchmarks).
Release Gate Workflow
This flowchart represents the sequential validation steps required for a production-grade release.
Sources: scripts/quality_gate.py:461-547
Configurable Thresholds
The gates utilize environment variables to define failure conditions, ensuring that performance or quality regressions block the CI/CD pipeline.
| Variable | Default (Strict) | Description |
|---|---|---|
QUALITY_EVAL_FAIL_WER_ABOVE | 0.10 | Maximum allowed WER on LibriSpeech test-clean. |
PERF_BENCH_FAIL_RTF_ABOVE | 0.50 | Maximum Real Time Factor (RTF) allowed. |
STREAMING_QUALITY_FAIL_REWRITE_RATE_ABOVE | 0.30 | Maximum allowed rewrite rate for streaming. |
Sources: scripts/quality_gate.py:126-140, scripts/quality_gate.py:513-520
Dataset Management
Evaluation scripts support automated fetching and processing of standard datasets.
- LibriSpeech: The
eval_librispeech.pyscript handles downloading from OpenSLR, extraction, and deterministic sampling (sequential or speaker round-robin). - Manifests: JSONL manifests are used for batch evaluation, mapping local
audio_pathentries toreference_text.
Sources: scripts/eval_librispeech.py:48-90, scripts/eval_manifest_quality.py:117-142
The evaluation tooling and metrics suite ensures that mlx-qwen3-asr maintains high fidelity to the original model architecture while optimizing for Apple Silicon performance. By combining traditional ASR metrics with parity analysis and streaming diagnostics, the project provides a robust framework for continuous quality assurance.