Async IO and io_uring Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Async IO and io_uring pipeline manages high-throughput expert streaming from disk to memory. It allows the Colibrì engine to run massive Mixture-of-Experts (MoE) models by overlapping disk reads with active computation. By utilizing an asynchronous I/O pool and the io_uring interface, the system hides storage latency and prevents the compute thread from stalling during expert retrieval.
Sources: README.md:144-149, SUMMARY.md:131-136
Architecture and Data Flow
The pipeline treats experts as data to be staged across a heterogeneous storage hierarchy. The engine activates only a small percentage of total parameters per token, streaming the required experts from NVMe storage into a per-layer LRU cache or VRAM tier.
The Streaming Process
- Router Lookahead: A pilot thread predicts required experts one layer ahead.
- Batch-Union: The system identifies unique experts required for a batch to avoid redundant reads.
- Async Fetch: The
PIPE=1pool initiates asynchronous reads for missing experts. - Compute Overlap: The engine performs matmul on resident experts while the I/O pool fetches the remainder.
- Placement: Fetched experts move into the managed memory hierarchy (VRAM/RAM).
Sources: README.md:80-92, README.md:144-152, README.it.md:71-85
I/O Pipeline Flow
The following diagram illustrates how the engine coordinates computation and asynchronous loading.
The diagram shows the lookahead mechanism triggering async fetches through io_uring to ensure data is staged before the compute phase.
Sources: README.md:111-120, README.it.md:71-78
Key Components and Controls
The pipeline behavior is controlled by specific environment variables and engine flags that dictate how the I/O pool interacts with the operating system.
| Variable | Type | Description |
|---|---|---|
PIPE | Boolean | Activates the bounded async I/O pool (default: 1). |
URING | Boolean | Enables io_uring batching for expert reads. |
PILOT | Boolean | Enables the router-lookahead thread for prefetching. |
DIRECT | Boolean | Uses O_DIRECT to bypass the OS page cache for NVMe. |
DISK_SPLIT | Boolean | Instrumenting flag to track which phase reads experts. |
Sources: README.md:144-155, c/tools/diag_harness.py:270-272, c/tests/test_efficiency_report.py:73-77
Performance Diagnostics
The engine provides detailed telemetry to measure the efficiency of the Async IO pipeline. Key metrics include the "felt wait" time, which measures the actual compute stall caused by disk I/O, and the "read service" time, which represents total time spent on I/O threads.
I/O Timing Breakdown
The diagnostic harness tracks these specific buckets to identify bottlenecks.
The sequence diagram represents the overlap between the Compute Thread and the I/O Pool, demonstrating how io_uring handles background fetches.
Sources: README.md:144-152, c/tools/diag_harness.py:302-310, SUMMARY.md:131-140
Efficiency Thresholds
The efficiency report utilizes specific thresholds to determine if the I/O pipeline is the primary bottleneck.
- Disk-Wait Dominant: If more than 40% of decode wall-time is spent waiting on experts, the system is considered I/O-bound.
- Felt Wait Ratio: If "felt wait" exceeds 50% of "read service" time, it indicates that the async pool is failing to overlap computation effectively.
- Direct I/O Win: On high-bandwidth NVMe drives,
DIRECT=1can improve decode speed by up to 34% by bypassing the page cache.
Sources: c/tests/test_efficiency_report.py:29-35, c/tests/test_efficiency_report.py:168-171, README.md:153-160
Implementation Details
The core I/O logic resides in c/colibri.c. Expert matrices are stored adjacently on disk to allow for a single pread operation per expert, minimizing syscall overhead. When URING=1 is active, multiple pread requests are batched into the io_uring submission queue.
Storage Hierarchy Interaction
The pipeline interacts with the learning cache (.coli_usage) to determine residency. The engine automatically pins the most frequently accessed experts based on measured routing heat, which reduces the number of async I/O requests over time.
Sources: README.md:126-140, README.it.md:104-109, SUMMARY.md:131-140
Async IO and the io_uring pipeline are critical for maintaining interactive speeds when running frontier models on consumer hardware. By shifting the bottleneck from memory capacity to disk bandwidth and overlapping retrieval with execution, Colibrì allows for execution of models that far exceed physical RAM limits.
Sources: README.md:12-16, README.it.md:12-16