Expert Streaming & Dual-SSD

Relevant source files

The following files were used as context for generating this wiki page:

• c/iobench.c
• c/tests/test_st_mirror.c
• c/tests/test_st.c
• README.md
• CHANGELOG.md
• c/inkling.c
• c/tools/efficiency.py

Expert Streaming and Dual-SSD functionality allow the Colibrì engine to run large Mixture-of-Experts (MoE) models on consumer hardware by treating storage as a managed memory tier. Instead of loading the entire model into VRAM or RAM, the engine streams routed experts from disk on demand. The Dual-SSD feature leverages multiple storage devices to increase aggregate bandwidth, effectively doubling the streaming throughput on disk-bound hosts.

Sources: README.md:29-45, CHANGELOG.md:57-60

Architecture and Memory Hierarchy

Colibrì manages a three-tier memory hierarchy: VRAM (Hot), RAM (Warm), and NVMe (Cold). The "dense" parts of the model, such as attention mechanisms and embeddings, remain resident in RAM. The 19,456 routed experts are stored on disk and staged into memory only when the router selects them for a specific token.

Memory Tier Roles

Tier	Residency	Content Type
VRAM	Hot	Pinned hottest experts, active expert activations.
RAM	Warm	Dense weights (attention, shared experts), LRU expert cache.
NVMe	Cold	Full expert set (e.g., ~370 GB for GLM-5.2).

Sources: README.md:104-125, c/inkling.c:50-65

Data Flow Diagram

Dual-SSD Streaming

The Dual-SSD feature (gated by COLI_MODEL_MIRROR) allows the engine to read model weights from two drives simultaneously. This is primarily used to overcome the bandwidth limitations of a single SATA or NVMe drive.

Functional Behavior

• Deterministic Routing: Each expert is assigned to a specific drive based on a deterministic hash of its identifier.
• Bandwidth Weighting: The engine balances the load based on measured or declared bandwidth ratios between the primary and mirror drives.
• Startup Validation: The engine validates the mirror at startup. The safetensors header and file sizes must be byte-identical to the primary drive.
• Fault Tolerance: If a read error occurs on the mirror drive, the engine falls back to the primary drive to prevent a system crash.

Sources: README.md:162-182, CHANGELOG.md:57-58, c/tests/test_st_mirror.c:10-40

Multi-Drive Shard Splitting

In addition to mirroring, the system supports N-drive shard splitting via COLI_MODEL_DIRS. This aggregates capacity by spreading a single container across multiple drives without duplication, allowing the engine to run models larger than any single available disk.

Sources: CHANGELOG.md:59-60

Implementation Details

Asynchronous I/O and Prefetching

The engine uses an asynchronous I/O pool (PIPE=1) to overlap computation with data transfer. While resident experts are computing, the PILOT=1 thread prefetches experts for the next layer. This prefetch mechanism is approximately 71.6% predictive.

Sources: README.md:200-210, c/inkling.c:495-505

Key Functions and Structures

Element	Description
COLI_MODEL_MIRROR	Environment variable that enables Dual-SSD streaming by specifying the mirror directory.
st_read_f32	Core function for reading tensor data, adapted for multi-shard and mirrored environments.
expert_load	Orchestrates the movement of expert weights from storage into the LRU or pinned cache slots.
Slot	Data structure representing a cached expert, including its ID, usage clock, and storage tier status.

Sources: c/inkling.c:85-105, c/tests/test_st.c:15-35, c/tools/efficiency.py:80-95

Diagnostic and Performance Monitoring

The diag_harness.py and efficiency.py tools track streaming performance. Specifically, they measure the "felt wait" (compute stalls caused by I/O) versus "read service" (actual time spent on I/O threads).

Configuration Options

Parameter	Type	Default	Description
DIRECT	Integer	0	Enables O_DIRECT to bypass the OS page cache for faster NVMe reads.
PIPE	Integer	1	Enables the asynchronous I/O pool for overlapping reads.
COLI_MODEL_MIRROR	Path	None	Path to the byte-identical model mirror on a second drive.
COLI_DISK_WEIGHTS	String	Measured	Manual bandwidth ratio for Primary,Mirror (e.g., "9,3").

Sources: README.md:162-166, README.md:213-220, c/tools/efficiency.py:160-170

The Expert Streaming and Dual-SSD systems are critical for maintaining Colibrì's "token-exact" validation while operating on limited hardware. By efficiently staging data across tiered storage, the project enables large-scale model inference without requiring enterprise-grade memory capacities.

Sources: README.md:223-228, CHANGELOG.md:73-80