Model Lifecycle & Session Management

Relevant source files

The following files were used as context for generating this wiki page:

• mlx_qwen3_asr/load_models.py
• mlx_qwen3_asr/session.py
• mlx_qwen3_asr/server.py
• mlx_qwen3_asr/transcribe.py
• tests/test_load_models.py
• CLAUDE.md

The Model Lifecycle and Session Management system in mlx-qwen3-asr provides a structured framework for loading, caching, and managing Qwen3-ASR models on Apple Silicon. It ensures memory efficiency through a multi-slot LRU cache and provides an explicit Session API for managing the model and tokenizer lifecycle during inference.

This system bridges the gap between raw model weights and high-level transcription requests by handling path resolution, weight remapping, quantization inference, and asynchronous execution.

Sources: CLAUDE.md:46-47, mlx_qwen3_asr/session.py, mlx_qwen3_asr/load_models.py:21-42

Model Loading Pipeline

The loading pipeline transforms a HuggingFace repository ID or a local directory into a functional Qwen3ASRModel instance. This process involves automatic downloading, configuration parsing, and weight remapping to match the MLX implementation.

Resolution and Initialization Flow

When a model is requested, the system first resolves the path. If the path does not exist locally with a config.json, it triggers a snapshot_download from the HuggingFace Hub. Once resolved, the system performs the following steps:

1. Config Parsing: Reads config.json to instantiate a Qwen3ASRConfig.
2. Weight Loading: Loads .safetensors files and applies remap_weights to convert PyTorch-style keys to MLX-native keys.
3. Tied Embedding Materialization: If the model uses tied LM heads, the system ensures the lm_head.weight is correctly mapped from the embedding weights.
4. Quantization Handling: If a quantization_config.json exists or quantized tensors are detected, the model architecture is modified using nn.quantize before weight loading.

Sources: mlx_qwen3_asr/load_models.py:59-106, mlx_qwen3_asr/load_models.py:212-225, tests/test_load_models.py:231-248

The flow shows the transition from a raw identifier to a loaded MLX model.
Sources: mlx_qwen3_asr/load_models.py:61-106

Session Management

The Session class acts as the primary interface for managing a model's operational lifecycle. It encapsulates both the Qwen3ASRModel and its associated Tokenizer, providing high-level methods for transcription and alignment.

Key Components

• Lazy Loading: Sessions do not necessarily load the model immediately upon instantiation, allowing for deferred memory allocation.
• Async Execution: The transcribe_async method offloads heavy inference tasks to prevent blocking the main event loop, which is critical for the built-in HTTP server.
• Context Management: Sessions maintain state required for stable transcription across multiple audio chunks.

Sources: mlx_qwen3_asr/session.py, mlx_qwen3_asr/server.py:126-145

Session API Methods

Method	Description
transcribe()	Synchronous high-level entry point for audio to text.
transcribe_async()	Asynchronous version of transcription, offloading to threads.
align()	Generates word-level timestamps for a given text and audio.
close()	Explicitly releases resources and clears associated caches.

Sources: mlx_qwen3_asr/session.py

Model Caching Strategy

To avoid the overhead of re-loading large models (0.6B or 1.7B parameters) during repeated calls, the project implements a _ModelHolder class that manages a process-local LRU cache.

The _ModelHolder Class

The cache is keyed by a tuple of (path_or_hf_repo, dtype). This allows different versions of the same model (e.g., a float16 version and a quantized version) to reside in memory simultaneously if needed.

• Capacity: By default, the cache holds up to 4 entries, but this is configurable via set_cache_capacity.
• Isolation: Models are isolated by their data type to prevent accidental reuse of a model with the wrong precision.
• Lifecycle: When the cache capacity is exceeded, the Least Recently Used (LRU) model is evicted to free system memory.

Sources: mlx_qwen3_asr/load_models.py:21-56, tests/test_load_models.py:251-325

This diagram illustrates how the _ModelHolder intercepts loading requests to provide cached instances.
Sources: mlx_qwen3_asr/load_models.py:32-42

Server-Side Lifecycle Integration

In the context of the mlx-qwen3-asr serve command, the model lifecycle is integrated into the FastAPI lifespan. The model is loaded once at startup into a Session and shared across all incoming requests.

Inference Locking

Because MLX execution on Metal is typically serialized for performance, the server uses an asyncio.Lock (referred to as inference_lock) to manage access to the model.

1. Job Queueing: Requests are placed in an asyncio.Queue (default depth of 10).
2. Sequential Processing: A background worker pulls jobs from the queue and acquires the inference_lock before calling the session's transcription methods.
3. Backpressure: If the queue is full, the server returns a 503 Service Unavailable error.

Sources: mlx_qwen3_asr/server.py:84-110, mlx_qwen3_asr/server.py:126-140, mlx_qwen3_asr/server.py:188-196

Quantization Inference

The system can automatically infer quantization parameters if weights are loaded from a source that was previously quantized using MLX but lacks a quantization_config.json.

The _infer_quantization_params function analyzes the shape of packed weights relative to the original model's dimensions:

• Bits: Calculated by comparing input_dim with packed_cols * 32. Supported values are 4 and 8.
• Group Size: Inferred by the ratio of input_dim to the number of columns in the scales tensor. Supported values include 32, 64, and 128.

Sources: mlx_qwen3_asr/load_models.py:183-225, tests/test_load_models.py:339-357

Summary

Model lifecycle management in mlx-qwen3-asr centers on the balance between performance and memory usage. By utilizing an LRU cache in _ModelHolder, the system minimizes loading latency for repeated tasks. The Session API provides a clean abstraction for users to interact with the models, while the server implementation demonstrates how to integrate these components into a high-concurrency environment using asynchronous patterns and inference locking.