Weight Conversion & Quantization
Relevant source files
The following files were used as context for generating this wiki page:
Weight conversion and quantization are critical processes in the mlx-qwen3-asr project, designed to adapt HuggingFace Qwen3-ASR model weights for high-performance inference on Apple Silicon using the MLX framework. This system handles the remapping of weight keys, transposition of convolutional layers, and the reduction of model precision through quantization to optimize memory usage and execution speed.
The pipeline ensures that the native Metal-optimized layers can correctly ingest weights originally trained in PyTorch. By supporting 4-bit and 8-bit quantization, the project enables large models (like the 1.7B variant) to run efficiently on devices with limited unified memory.
Sources: CLAUDE.md, scripts/convert.py:1-15, mlx_qwen3_asr/convert.py:1-10
Weight Remapping & Adaptation
The conversion process begins by remapping the keys and shapes of HuggingFace safetensors to match the internal structure of the MLX implementation. This is necessary because the original weights often contain prefixes or formats incompatible with the MLX nn.Module hierarchy.
Key Transformation Logic
The primary remapping involves stripping the thinker. prefix from HuggingFace weight keys. For example, thinker.audio_tower.conv2d1.weight is transformed to audio_tower.conv2d1.weight.
Conv2d Transposition
MLX and PyTorch utilize different memory layouts for convolutional layer weights. During conversion, weights for the Conv2d stem are transposed from the PyTorch format (out_channels, in_channels, kH, kW) to the MLX format (out_channels, kH, kW, in_channels) using the transformation transpose(0, 2, 3, 1).
Sources: mlx_qwen3_asr/convert.py:10-48
This diagram shows the transformation flow from source weights to MLX-compatible tensors.
Sources: mlx_qwen3_asr/convert.py:27-46
Quantization Strategy
Quantization reduces the precision of model weights (typically from float16 to 4 or 8 bits) to decrease the memory footprint and potentially improve inference latency on Metal.
Quantization Parameters
The system supports two main quantization levels and allows for configurable group sizes to balance accuracy and performance.
| Parameter | Supported Values | Description |
|---|---|---|
bits | 4, 8 | Number of bits per weight. |
group_size | 32, 64, 128 | The number of weights that share a quantization scale. |
dtype | float16, float32, bfloat16 | The intermediate precision used before quantization. |
Sources: scripts/convert.py:35-58, mlx_qwen3_asr/load_models.py:214-220
Linear and Embedding Quantization
Quantization is primarily applied to Linear and Embedding layers. When loading a model, the system can automatically infer quantization parameters by checking for .scales suffixes in the weight dictionary. If .scales keys are present, the model's nn.Module structure is modified using nn.quantize before the weights are loaded.
Sources: mlx_qwen3_asr/convert.py:51-68, mlx_qwen3_asr/load_models.py:114-124
Conversion Workflow
The conversion is typically performed via the scripts/convert.py utility, which orchestrates the loading, remapping, quantization, and saving of the model.
This sequence illustrates the lifecycle of a weight conversion and quantization task.
Sources: scripts/convert.py:65-120
Automated Publishing
The scripts/publish_quantized.py script automates the creation of quantized artifacts for distribution on the HuggingFace Hub. It performs the following steps:
- Creates a temporary directory.
- Runs the conversion script with the specified bit-width and group size.
- Generates a
README.mdandquantization_config.jsonfor the target repository. - Uploads the resulting MLX-native folder to HuggingFace.
Sources: scripts/publish_quantized.py:40-105
Performance Benchmarking
The project includes scripts/eval/benchmark_quantization_matrix.py to evaluate the trade-offs between different quantization configurations. This script generates a matrix comparing Word Error Rate (WER), Character Error Rate (CER), and Real-Time Factor (RTF) across various bit-widths and group sizes.
Benchmarking Metrics
- Latency Mean: Average time taken for transcription.
- RTF (Real-Time Factor): Ratio of processing time to audio duration.
- WER/CER: Accuracy metrics relative to a reference transcription.
Sources: scripts/eval/benchmark_quantization_matrix.py:1-25, scripts/eval/benchmark_quantization_matrix.py:180-200
Summary
Weight conversion and quantization in mlx-qwen3-asr serve as the bridge between standard HuggingFace models and optimized Metal-based inference. By handling structural remapping and bit-width reduction, the system allows Qwen3-ASR models to be deployed effectively on Apple Silicon hardware with minimal loss in transcription quality.
Sources: CLAUDE.md, scripts/README.md