Quality Gates & Validation
Relevant source files
The following files were used as context for generating this wiki page:
Quality Gates & Validation represent a critical infrastructure within the mlx-qwen3-asr repository designed to ensure architectural correctness, performance stability, and transcription accuracy. These gates act as automated checkpoints that must be passed before Pull Requests are merged or new releases are published. The system spans from basic linting and unit tests to complex multilingual parity benchmarks against reference PyTorch implementations.
The validation suite is categorized into "Fast" and "Release" modes. While the fast mode focuses on code health (static analysis and unit tests), the release mode executes deep evaluations including Word Error Rate (WER) and Character Error Rate (CER) regressions, streaming stability metrics, and hardware-specific performance benchmarks on Apple Silicon.
Sources: scripts/quality_gate.py:10-14, CLAUDE.md:120-130
Quality Gate Execution Framework
The primary entry point for validation is scripts/quality_gate.py. This script orchestrates a series of StepResult objects, each representing an individual validation task.
Operational Modes
The quality gate supports two distinct operational modes:
| Mode | Scope | Purpose |
|---|---|---|
| Fast | Linting (Ruff), Type Checking (Mypy), Unit Tests (Pytest). | Rapid developer feedback before commits. |
| Release | All "Fast" steps + Reference Parity, LibriSpeech Quality, Manifest Quality, Performance Benchmarks, and Streaming Evaluation. | Comprehensive validation before merging to main or publishing to PyPI. |
Sources: scripts/quality_gate.py:596-601, CLAUDE.md:120-125
Validation Workflow
The following diagram illustrates how the run_gate function orchestrates the sequence of validation steps based on the selected mode.
Sources: scripts/quality_gate.py:417-585
Core Validation Components
Static Analysis and Type Safety
The project enforces strict type safety on core architectural components. Mypy is configured to target critical modules such as the attention mechanism, encoder, decoder, and model glue code.
Sources: scripts/quality_gate.py:18-25
Reference Parity Analysis
A key requirement for the MLX implementation is matching the output of the original PyTorch qwen-asr. The validation system categorizes mismatches into specific buckets to identify if differences are cosmetic or represent logical regressions.
| Category | Definition |
|---|---|
| exact_match | Identical token output. |
| punctuation_or_tokenization | Differences only in non-alphanumeric characters. |
| numeric_surface_form | Differences in digit representation (e.g., "53" vs "fifty-three"). |
| minor_lexical_shift | Edit distance ratio ≤ 0.12. |
| content_shift | Significant differences in transcribed text. |
Sources: scripts/eval/analyze_reference_parity_mismatches.py:34-53
Transcription Quality Metrics
The project utilizes several evaluation scripts to calculate accuracy across different datasets.
- LibriSpeech Evaluation: Uses deterministic sampling (sequential or speaker round-robin) to calculate WER and CER on standard English benchmarks. Sources: scripts/eval_librispeech.py:125-150
- Manifest Quality: Evaluates multilingual manifests, automatically switching between WER and CER based on whether the language is character-primary (e.g., Chinese, Japanese, Korean). Sources: scripts/eval_manifest_quality.py:58-75, scripts/eval_manifest_quality.py:220-235
Streaming Validation
Streaming quality is measured through diagnostic metrics that track the stability of partial transcriptions as audio chunks are processed.
The streaming gate fails if metrics fall below defined thresholds (e.g., partial_stability < 0.85 or rewrite_rate > 0.30).
Sources: scripts/quality_gate.py:118-164, scripts/eval_streaming_manifest.py:126-150
Performance and Hardware Validation
As a package optimized for Apple Silicon, performance benchmarking is a non-negotiable gate. The perf-benchmark step measures:
- RTF (Real Time Factor): Must be below 0.50 for strict release gates.
- Latency: Mean latency must stay within defined thresholds (defaulting to 2.0s for strict releases).
- Throughput: Benchmarked across 0.6B and 1.7B model variants.
Sources: scripts/quality_gate.py:59-100, CLAUDE.md:25-30
Conclusion
The Quality Gates & Validation system in mlx-qwen3-asr ensures that the "DNA" of the Qwen3-ASR architecture is preserved during its transition to a native MLX implementation. By combining strict architectural invariants with automated performance and accuracy regressions, the project maintains production-grade stability on macOS. Passing the full release mode gate is a prerequisite for any PyPI distribution, ensuring that users receive a validated, high-performance speech recognition engine.