Qwen3-ASR Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
The Qwen3-ASR project is a ground-up MLX reimplementation of the Qwen3-ASR speech recognition model, optimized for Apple Silicon. It is designed as an encoder-decoder transformer model that translates audio input into text. The architecture is built entirely using mlx.nn.Module to ensure native performance on Metal without relying on PyTorch wrappers.
Sources: CLAUDE.md:8-20, mlx_qwen3_asr/model.py:1-12
Core System Architecture
The Qwen3-ASR architecture follows a classic encoder-decoder transformer paradigm specifically adapted for multimodal speech-to-text tasks. The system processes raw audio into a mel spectrogram, encodes these features, and injects them into a text decoder sequence for autoregressive generation.
High-Level Components
| Component | Description |
|---|---|
| Audio Tower (Encoder) | Processes mel spectrograms through a Conv2d stem and transformer layers to produce audio features. |
| Text Decoder | A Qwen3-style transformer that generates text tokens based on embedded text and injected audio features. |
| LM Head | A linear projection layer that maps decoder hidden states to vocabulary logits. |
| KV Cache | Manages key-value states for efficient autoregressive decoding. |
Sources: mlx_qwen3_asr/model.py:44-55, CLAUDE.md:73-81
Data Flow Transformation
The following diagram illustrates the transformation of data from raw audio to final text logits.
The workflow ensures that audio features are properly aligned with text embeddings at specific <|audio_pad|> positions.
Sources: CLAUDE.md:73-81, mlx_qwen3_asr/model.py:126-145
Audio Encoder (Audio Tower)
The Audio Tower is responsible for extracting high-dimensional features from acoustic inputs. It utilizes a convolutional stem followed by a series of transformer layers (24 layers for the 1.7B model and 18 for the 0.6B model).
- Conv2d Stem: Performs an 8x downsampling of the mel spectrogram.
- Positional Embedding: Uses Sinusoidal Position Embeddings, which are computed dynamically rather than loaded from weights.
- Encoder Layers: Employs windowed attention to manage long audio sequences efficiently.
Sources: CLAUDE.md:73-77, mlx_qwen3_asr/model.py:44-49, tests/test_config.py:34-60
Text Decoder and Audio Injection
The text decoder handles the generation process. A critical aspect of this architecture is the Audio Injection mechanism, which replaces placeholder audio tokens in the text sequence with features generated by the Audio Encoder.
Injection Logic
The system identifies audio_token_id positions within the input_ids. It uses cumulative-sum indexing to map each placeholder position to the corresponding audio feature vector, selecting the values via mx.where.
# mlx_qwen3_asr/model.py:173-178
audio_counts = mx.sum(audio_mask.astype(mx.int32), axis=1)
max_count = int(mx.max(audio_counts).item())
# ...
cum_idx = mx.cumsum(audio_mask.astype(mx.int32), axis=1) - 1
cum_idx = mx.maximum(cum_idx, 0)
Sources: mlx_qwen3_asr/model.py:150-198
Text Decoder Features
- MRoPE (Multi-RoPE): Interleaved, not chunked, using a stride-3 frequency assignment across temporal, height, and width dimensions.
- RMSNorm: The decoder uses RMSNorm and lacks biases in the linear layers, contrasting with the Encoder which uses LayerNorm with bias.
Sources: CLAUDE.md:83-90, mlx_qwen3_asr/model.py:51-60
Transcription Pipeline
The high-level transcription process is managed by mlx_qwen3_asr/transcribe.py. It coordinates audio loading, chunking for long files, and the interaction between the model components.
Sequence of Operations
The transcription sequence is depicted below:
Configuration and Model Variants
The project supports two main model sizes: 0.6B and 1.7B. These are configured through specific dataclasses that define hidden sizes, layer counts, and attention head configurations.
| Metric | 0.6B Model | 1.7B Model |
|---|---|---|
| Encoder Layers | 18 | 24 |
| Decoder Layers | 28 | 28 |
| D Model (Encoder) | 896 | 1024 |
| Hidden Size (Decoder) | 1024 | 2048 |
| Vocab Size | 151936 | 151936 |
Sources: tests/test_config.py:34-60, tests/test_config.py:99-130
Summary
The Qwen3-ASR architecture is a sophisticated multimodal system designed for high-performance speech recognition on Apple Silicon. By combining a convolutional-transformer audio encoder with a robust text decoder using native MLX operations, it achieves low-latency transcription while maintaining high accuracy across 52 supported languages. The implementation emphasizes technical correctness in areas like MRoPE interleaving and weight mapping to ensure parity with the original HuggingFace weights.