KV-Cache Streaming
Relevant source files
The following files were used as context for generating this wiki page:
KV-Cache Streaming is an experimental feature in mlx-qwen3-asr that enables linear-complexity incremental decoding for automatic speech recognition (ASR). It allows the system to process audio streams in real-time or near-real-time by reusing Key-Value (KV) caches across successive audio chunks, significantly reducing the computational overhead compared to re-processing the entire audio history for every new segment.
The system incorporates advanced logic for context trimming, tail refinement, and CJK-aware (Chinese, Japanese, Korean) stable/unstable text splitting. This ensures that transcription remains accurate and responsive, even during long-duration audio capture or live microphone input.
Sources: CLAUDE.md:37-38, mlx_qwen3_asr/streaming.py:1-5
Architecture and Core Logic
The streaming architecture is built around a persistent StreamingState object that tracks audio buffers, accumulated history, and the state of the incremental decoder.
Streaming Lifecycle
The process follows a three-stage lifecycle: initialization, iterative audio feeding, and finalization.
The following diagram illustrates the internal data flow during the feed_audio cycle:
Key Components
StreamingState
A dataclass used to maintain the session state. It stores both the raw audio data and the metadata required for incremental MLX inference.
| Attribute | Type | Description |
|---|---|---|
buffer | np.ndarray | Pending audio samples not yet processed into chunks. |
audio_accum | np.ndarray | History of samples used for context and tail refinement. |
stable_text | str | Transcription segments considered finalized and unlikely to change. |
_cache | object | The MLX KV-cache object reused across incremental steps. |
_next_position | int | The current position index for RoPE/positional embeddings. |
Sources: mlx_qwen3_asr/streaming.py:46-85
Incremental Decoding Engine
The engine utilizes a specific MLX runtime to avoid redundant computation. If the context exceeds max_context_sec, the system performs a reset to prevent unbounded cache growth and "stale-context drift."
_ensure_stream_runtime: Resolves the model and tokenizer for the session._decode_chunk_incremental: Processes a single chunk by building prompts (either initial or follow-up) and invoking the model'sprefillandstepmethods._decode_tokens_incremental: An autoregressive loop that generates tokens until an EOS (End Of Silence/Speech) or a repetition is detected.
Sources: mlx_qwen3_asr/streaming.py:246-340
Endpointing Strategies
The system supports two primary modes for determining when an audio chunk should be processed:
- Fixed: Processes audio in rigid increments defined by
chunk_size_sec(default 2.0s). - Energy: Searches for low-energy (silence) boundaries within a lookback window near the fixed boundary. This minimizes word clipping by aligning chunk breaks with natural pauses in speech.
Text Stability and Refinement
Stable/Unstable Splitting
To provide a smooth user experience, the system distinguishes between "stable" text (finalized) and "unstable" text (subject to change as more context arrives).
- CJK Awareness: Splitting logic treats Chinese, Japanese, and Korean differently, often splitting by character rather than whitespace.
- Unfixed Buffers: Uses
UNFIXED_CHUNK_NUM(default 2) andUNFIXED_TOKEN_NUM(default 5) to determine the rollback safety margin.
Sources: mlx_qwen3_asr/streaming.py:27-31, mlx_qwen3_asr/streaming.py:441-465
Tail Refinement
When streaming finishes, there may be residual audio in the buffer. The finish_streaming function performs a final decode. If this produces no progress, it triggers a "Tail Refinement" using a bounded window of the most recent audio history to recover potentially missed speech.
Sources: mlx_qwen3_asr/streaming.py:202-230, tests/test_streaming.py:288-310
Configuration Options
Streaming parameters can be configured via init_streaming or CLI arguments.
| Parameter | Default | Description |
|---|---|---|
chunk_size_sec | 2.0 | Duration of audio processed in one incremental step. |
max_context_sec | 30.0 | Maximum audio history retained for KV-cache context. |
endpointing_mode | "fixed" | "fixed" or "energy" based chunking. |
finalization_mode | "accuracy" | Policy for the final tail: "accuracy" (uses refinement) or "latency". |
unfixed_token_num | 5 | Number of trailing tokens kept in the "unstable" state. |
Sources: mlx_qwen3_asr/streaming.py:88-123, mlx_qwen3_asr/cli.py:530-555
Conclusion
KV-Cache Streaming provides a high-performance path for real-time transcription on Apple Silicon. By combining incremental MLX inference with intelligent endpointing and tail refinement, the system balances the competing needs of low latency and high transcription accuracy. This module is essential for features such as live microphone transcription and processing long-form audio files without excessive memory usage.
Sources: CLAUDE.md:37-38, mlx_qwen3_asr/streaming.py:236-243