Introduction & Project North Star
Relevant source files
The following files were used as context for generating this wiki page:
The mlx-qwen3-asr project is a ground-up MLX reimplementation of the Qwen3-ASR model, specifically optimized for Apple Silicon. Its primary purpose is to provide a definitive, standalone speech recognition package that leverages Metal for high-performance inference without relying on heavy external dependencies like PyTorch or HuggingFace Transformers at runtime.
The project's "North Star" is to achieve a frictionless user experience: pip install mlx-qwen3-asr. It aims to be the standard ASR solution for macOS users by offering one-command setup, native forced alignment for word-level timestamps, and a built-in transcription server, all while maintaining strict architectural parity with the original Qwen3-ASR weights.
Sources: CLAUDE.md:5-24, pyproject.toml:1-25
Core Objectives and Principles
The project is governed by several critical technical decisions and constraints designed to ensure speed, accuracy, and maintainability on macOS.
The "Must-Have" Feature Set
- Metal-Optimized Inference: Sub-200ms latency on short clips using the MLX framework.
- Zero Transformers Dependency: Uses a custom mel spectrogram and native BPE tokenizer to avoid
transformersat runtime. - No PyTorch in Core Path: The pipeline from audio to text (audio → mel → encoder → decoder → text) must never import
torch. - Native Forced Aligner: Word-level timestamps are provided via a native MLX implementation rather than PyTorch bindings.
Sources: CLAUDE.md:9-21, CLAUDE.md:120-135
Architecture Overview
Qwen3-ASR follows an encoder-decoder architecture. The audio is converted to a mel spectrogram (128 bins) and passed through a Conv2d stem (8x downsample) followed by 24 transformer encoder layers. These audio features are then injected into a text decoder (28 Qwen3-style layers) at specific <|audio_pad|> positions to generate text autoregressively.
Logic flow showing the transition from raw audio to transcribed text.
Sources: CLAUDE.md:96-105
Technical Constraints and Correctness Invariants
To maintain parity with the original model weights, the implementation must adhere to specific architectural invariants. Violating these produces silent errors where the model runs but generates incorrect output.
| Invariant | Implementation Detail |
|---|---|
| MRoPE | Interleaved sections [24,20,20] with stride-3 frequency assignment. |
| Normalization | Encoder uses LayerNorm + bias; Decoder uses RMSNorm with no bias. |
| Weight Transpose | Conv2d weights must be transposed from PyTorch (out,in,kH,kW) to MLX (out,kH,kW,in). |
| Embeddings | Sinusoidal position embeddings must be computed at runtime, not loaded from weights. |
| Tokens | Specific IDs: Pad=151676, Start=151669, End=151670. |
Sources: CLAUDE.md:107-118
Project Roadmap and Current Status
The project reached v0.1.0 with a stable core pipeline and comprehensive benchmarking against PyTorch (where MLX achieved 15.99% WER vs 16.69% for PyTorch).
Development Workflow
The project uses a structured memory workflow for continuous learning, where meaningful architectural decisions or failed paths are documented in docs/memory/.
The release process and quality checks required for publishing.
Sources: CLAUDE.md:46-80, AGENTS.md:10-15, scripts/update_readme_stats.py:10-50
User Interface and Integration
The primary entry point for users is the CLI, which supports various modes including offline transcription, streaming, and a local API server.
CLI Subcommands
- transcribe: The main entry point for processing audio files.
- serve: Launches a FastAPI-based HTTP server for remote transcription requests.
- doctor: Performs environment diagnostics to check for
ffmpeg,mlx, and optional dependencies likepyannote.audio.
Sources: mlx_qwen3_asr/cli.py:150-250, mlx_qwen3_asr/cli.py:340-360
Streaming Logic
Streaming is handled via KV-cache management with context trimming and tail refinement to maintain stability over long durations.
Interaction flow for the experimental streaming and microphone capture features.
Sources: mlx_qwen3_asr/cli.py:590-640, scripts/eval_streaming_manifest.py:170-200
Summary
mlx-qwen3-asr represents a specialized effort to bring high-quality Qwen3-ASR capabilities to the Apple Silicon ecosystem. By focusing on native MLX implementation and removing heavy dependencies like PyTorch from the core path, the project provides a performant, production-ready tool for local speech-to-text tasks, ranging from simple CLI transcription to complex, diarized meeting analysis.