Audio Processing & Mel Spectrograms

Relevant source files

The following files were used as context for generating this wiki page:

• mlx_qwen3_asr/audio.py
• scripts/generate_mel_filters.py
• scripts/eval_mel_parity.py
• mlx_qwen3_asr/cli.py
• tests/test_audio.py
• CLAUDE.md

The audio processing pipeline in mlx-qwen3-asr is a ground-up MLX reimplementation designed for high-performance speech recognition on Apple Silicon. It provides a native, standalone path for converting raw audio signals into log-mel spectrograms, which serve as the primary input for the Qwen3-ASR transformer encoder. This implementation eliminates dependencies on heavy frameworks like transformers or librosa for core inference tasks.

The system handles diverse audio formats via an integrated loading strategy that prioritizes a fast-path for WAV files while falling back to ffmpeg for compressed media. The resulting features are Whisper-compatible, 128-bin Slaney-normalized mel spectrograms, ensuring parity with official Qwen3-ASR and Whisper weights.

Audio Loading and Preprocessing

The project utilizes a multi-tiered approach to audio ingestion, ensuring compatibility with various input types including file paths, numpy arrays, and raw byte streams.

Audio Ingestion Logic

The primary entry point is load_audio_np, which canonicalizes inputs into a mono, float32 numpy array sampled at 16,000 Hz. The pipeline supports:

• Fast-path WAV loading: Uses standard library parsing for uncompressed PCM (format 1) and IEEE float (format 3) WAV files to avoid subprocess overhead.
• FFmpeg fallback: Handles compressed formats (AAC, MP3, FLAC, etc.) by spawning an ffmpeg subprocess to decode audio into 16-bit PCM.
• Resampling: If the source sample rate differs from the target 16kHz, the system pipes the audio through ffmpeg for high-quality resampling.

Key Audio Parameters

Parameter	Value	Description
SAMPLE_RATE	16000	Target frequency for ASR processing.
N_FFT	400	FFT window size (25ms at 16kHz).
HOP_LENGTH	160	Stride between STFT frames (10ms).
NUM_MEL_BINS	128	Number of frequency bins in the Mel filterbank.

Sources: mlx_qwen3_asr/audio.py:21-25, CLAUDE.md:120

Mel Spectrogram Computation

The feature extraction process converts raw time-domain waveforms into log-mel spectrograms. This is implemented natively in MLX to leverage GPU acceleration on Metal.

Feature Extraction Pipeline

The log_mel_spectrogram function performs the following steps:

1. Short-Time Fourier Transform (STFT): Applies a Hann window and computes the RFFT via mx.as_strided for memory-efficient framing. It uses reflect padding to match Whisper/librosa conventions.
2. Power Spectrogram: Calculates the magnitude squared of the complex STFT output.
3. Filterbank Projection: Multiplies the power spectrogram by a pre-computed 128-bin Slaney-normalized mel filterbank.
4. Log-Scaling: Applies a base-10 logarithm with clamping at a -8.0 dB floor relative to the maximum value, followed by normalization (x + 4.0) / 4.0.

Mel Filterbank Generation

The project includes a standalone script scripts/generate_mel_filters.py to recreate the filterbank assets. It implements Slaney mel-scale conversion and area normalization in pure NumPy, ensuring the resulting mel_filters.npz provides accurate frequency weighting.

# scripts/generate_mel_filters.py:41-47
def mel_filterbank(n_mels: int = 128, n_fft: int = 400, sample_rate: int = 16000) -> np.ndarray:
    n_freqs = 1 + n_fft // 2
    fft_freqs = np.linspace(0.0, sample_rate / 2.0, n_freqs, dtype=np.float64)
    min_mel = _hz_to_mel_slaney(np.array([0.0], dtype=np.float64))[0]
    max_mel = _hz_to_mel_slaney(np.array([sample_rate / 2.0], dtype=np.float64))[0]

Sources: scripts/generate_mel_filters.py:41-75

Validation and Parity

To ensure the native MLX implementation produces identical results to the reference PyTorch implementation, the repository includes comprehensive parity testing.

Mel Parity Evaluation

The script scripts/eval_mel_parity.py compares the custom MLX frontend against the WhisperFeatureExtractor from the HuggingFace transformers library. It validates:

• Mean Absolute Error (MAE): Typically constrained to < 1e-5.
• Maximum Absolute Difference: Constrained to < 2e-4.
• Shape Consistency: Ensuring frame counts match exactly for various audio lengths.

Sources: scripts/eval_mel_parity.py:70-80, scripts/eval_mel_parity.py:100-105

Padding Modes

The compute_features function supports three padding modes to accommodate different inference requirements:

• do_not_pad: Returns the natural frame length.
• longest: Alias for natural length in single-audio contexts.
• max_length: Pads short clips to exactly 3000 frames (30 seconds) while maintaining the true feature_lens metadata for the model's encoder.

Sources: mlx_qwen3_asr/audio.py:327-338, tests/test_audio.py:317-335

Environment Diagnostics

The CLI includes a "doctor" command to verify that the host environment has the necessary audio processing dependencies installed.

# cli.py:149
mlx-qwen3-asr --doctor

This checks for:

1. FFmpeg binary: Required for non-WAV formats.
2. MLX Library: Required for feature computation.
3. Diarization Extras: (Optional) pyannote.audio and torch for speaker labeling.

Sources: mlx_qwen3_asr/cli.py:149-195

Summary

The audio processing subsystem provides a robust, Metal-optimized foundation for ASR. By implementing a custom STFT and Mel projection in MLX, the project achieves sub-200ms latency on short clips while maintaining strict mathematical parity with reference implementations. The system's ability to handle raw arrays or file-based inputs through a unified API makes it suitable for both production servers and real-time microphone applications.