Audio Encoder Component

Relevant source files

The following files were used as context for generating this wiki page:

• mlx_qwen3_asr/encoder.py
• mlx_qwen3_asr/transcribe.py
• mlx_qwen3_asr/audio.py
• mlx_qwen3_asr/model.py
• CLAUDE.md

The Audio Encoder Component is a critical subsystem of the mlx-qwen3-asr architecture, responsible for transforming raw audio signals into high-dimensional feature representations. It operates as the "front-end" of the encoder-decoder transformer model, processing 128-bin Mel spectrograms through a convolutional stem followed by a series of transformer encoder layers.

Within the project's "Metal-first" philosophy, this component is implemented entirely using MLX to ensure optimal performance on Apple Silicon. It handles the initial feature extraction (8x downsampling) and contextual modeling of audio frames before they are injected into the text decoder for transcription.

Sources: CLAUDE.md:10-25, mlx_qwen3_asr/encoder.py

Architecture Overview

The encoder architecture consists of three primary stages: feature preparation, a convolutional stem for downsampling, and a stack of Transformer Encoder layers.

Component Hierarchy

The encoder stack processes a 128-bin Mel spectrogram input and outputs refined audio features for the text decoder.
Sources: CLAUDE.md:88-92, mlx_qwen3_asr/encoder.py

Key Specifications

Attribute	Value	Source
Input Dimensions	128 Mel bins	CLAUDE.md:89
Downsampling Factor	8x (via Conv2d stem)	CLAUDE.md:90
Layer Count	24 Transformer Encoder Layers	CLAUDE.md:90
Weight Format	Metal-optimized MLX (transposed from PyTorch)	CLAUDE.md:102
Normalization	LayerNorm with bias	CLAUDE.md:101

Feature Extraction Pipeline

The pipeline begins with the transformation of raw 16kHz mono audio into a Mel spectrogram. This is a critical invariant; the system uses a custom MLX implementation to maintain a zero-dependency path.

1. Mel Spectrogram: Audio is converted to 128 bins.
2. Conv2d Stem: The spectrogram is passed through a convolutional stem that reduces the temporal resolution by a factor of 8.
3. Weight Transposition: To maintain parity with HuggingFace weights, convolutional weights are transposed from PyTorch format (out, in, kH, kW) to MLX format (out, kH, kW, in).

Sources: CLAUDE.md:90, CLAUDE.md:102, mlx_qwen3_asr/audio.py, mlx_qwen3_asr/encoder.py

Transformer Encoder Layers

The core of the component is a stack of 24 transformer layers. Unlike the text decoder (which uses RMSNorm), the audio encoder strictly utilizes LayerNorm with bias.

Windowed Attention

The encoder utilizes windowed attention to process long audio sequences efficiently. This allows the model to maintain local context while limiting computational complexity.

Positional Embeddings

The encoder uses computed Sinusoidal Position Embeddings. These are not loaded from weight files but are generated at runtime to provide temporal context to the transformer layers.

The AudioEncoder class manages the high-level flow, while individual EncoderLayer instances handle attention and normalization.
Sources: CLAUDE.md:90, CLAUDE.md:101-103, mlx_qwen3_asr/encoder.py

Data Flow and Integration

The Qwen3ASRModel acts as the orchestrator, passing data through the audio_tower (the encoder) before merging the results into the text embedding sequence.

Audio Injection Logic

Audio features are not simply appended; they are injected into the text decoder sequence at specific <|audio_pad|> placeholder positions. The encoder ensures the output matches the expected dimensionality of the text decoder's embedding space.

Implementation Constraints

To ensure correctness and parity with the official Qwen3-ASR implementation, the following invariants are enforced within the encoder:

• Normalization Divergence: The encoder MUST use LayerNorm + bias. Mixing this with the decoder's RMSNorm (which has no bias) will cause silent inference failure.
• No Torch dependency: The core encoder pipeline (Mel → Conv → Transformer) must never import torch, relying entirely on mlx.nn.Module.
• Sinusoidal PE: Positions must be computed mathematically, as they are absent from the safetensors weight files.

Sources: CLAUDE.md:13-20, CLAUDE.md:100-104

Summary

The Audio Encoder Component provides the foundational acoustic modeling for the Qwen3-ASR system. By utilizing a 24-layer transformer stack and a convolutional downsampling stem, it effectively compresses raw audio into a format suitable for the text decoder. Its implementation in MLX ensures that these heavy computations are performed natively on Metal, providing the high speed and low latency required for production-grade ASR on Apple Silicon.

Sources: CLAUDE.md:15-18, mlx_qwen3_asr/encoder.py