Memory Hierarchy and Tiering
Relevant source files
The following files were used as context for generating this wiki page:
Colibri implements a unified memory hierarchy to run large-scale Mixture-of-Experts (MoE) models on consumer hardware. It treats VRAM, RAM, and NVMe storage as a single managed storage pool for model weights. The engine stages parameters across these tiers based on routing frequency and latency requirements.
Sources: README.md:14-17, site/index.html:151-155
Tiered Architecture Overview
The system divides model parameters into two primary categories: resident dense weights and dynamic sparse experts. Dense weights, including attention mechanisms and embeddings, remain resident in high-speed memory. In contrast, the 19,456 routed experts are staged across a three-tier hierarchy based on measured routing "heat."
Sources: README.md:104-109, site/index.html:151-155
Memory Tiers
The following table describes the three primary tiers used in the colibri engine:
| Tier | Hardware | Description |
|---|---|---|
| VRAM | GPU Memory | Optional highest-speed tier for the most frequently routed experts. |
| RAM | System Memory | Holds the 17B dense parameters and "warm" experts in a pinned hot-store or LRU cache. |
| Disk | NVMe Storage | The "cold" base tier where all experts reside (~370 GB) and stream from on demand. |
Sources: README.md:104-109, site/index.html:151-155, c/telemetry.h:111-115
Component Distribution
The diagram below illustrates how model components are distributed across the memory hierarchy.
Expert Placement Logic
Colibri utilizes "Measured Routing Heat" to determine the optimal placement for each expert. The engine monitors which experts are activated by specific workloads and persists this data in a .coli_usage file.
Sources: README.md:113-116, c/telemetry.h:182-192
The Learning Cache
The placement system functions like a Just-In-Time (JIT) compiler for weights. It tracks routing predictability—noting that routing is approximately 71.6% predictable one layer ahead—to prefetch weights and hide staging latency.
- Usage Recording: The engine records expert routing in the
eusagearray. Sources: c/inkling.c:89, c/telemetry.h:138-140 - Frequency Analysis: A higher bit-count in the heat value (up to 63) indicates a "hotter" expert. Sources: c/telemetry.h:151-153, web/src/Brain.tsx:96-102
- Tier Promotion: Experts with high usage frequency are promoted from Disk to RAM (pinned) or VRAM. Sources: README.md:113-116, c/inkling.c:420-425
Data Structures for Tiering
The Slot and LCache structures in c/inkling.c manage the expert lifecycle:
typedef struct {
int eid; uint64_t used;
int pinned; /* never evicted */
int filled; /* 0 while queued for fill */
uint8_t *p13, *p2; float *s13, *s2; /* container: packed rows + scales */
// ...
} Slot;
typedef struct { Slot *slots; int n, cap; } LCache;
Sources: c/inkling.c:80-86
Staging and Prefetching
Staging experts from disk to RAM involves a five-step path for every token: Route, Union, Place, Overlap, and Learn. The engine uses a bounded async I/O pool (PIPE=1) to load missing experts while resident ones execute compute tasks.
Sources: README.md:140-146, site/index.html:153-155
Prefetch Pipeline
The PILOT=1 thread runs one layer ahead of the current computation to identify needed experts. By identifying the "union" of unique experts required for a batch of tokens, the engine minimizes redundant disk reads.
Implementation Details
Metal Residency Sets (macOS)
On Apple Silicon, Colibri uses MTLResidencySet to manage GPU memory residency for experts. This mechanism allows the engine to declare residency for a set of buffers once at the command queue level, rather than per command buffer, reducing overhead during MoE execution.
Sources: SUMMARY.md:14-22, SUMMARY.md:52-58
Quantization and Row Packing
To optimize storage and transfer, experts are stored in a packed format:
- int4 (fmt=4): Experts are stored as 4-bit nibbles, effectively doubling the capacity of the RAM cache.
- int8: Used for MTP (Medusa Tree Parity) heads to maintain acceptance rates.
- Row Scales: Every row in a quantized tensor includes a 32-bit floating-point scale.
Sources: c/inkling.c:75-79, README.md:172-178, c/telemetry.h:9-13
Configuration and Monitoring
Users can control the tiering behavior through environment variables that set memory budgets:
| Variable | Type | Description |
|---|---|---|
RAM_GB | Float | Maximum system RAM allocated for the expert cache. |
CUDA_EXPERT_GB | Float/Auto | VRAM budget for resident experts. |
PIN_GB | Float/All | Amount of memory to dedicate to the learned "hot-store." |
COLI_NUMA | Binary | Enables interleaving resident weights across memory controllers. |
Sources: README.md:116-118, site/index.html:190-195, c/inkling.c:380-385
Monitoring Tiers
The telemetry system emits TIERS and EMAP lines to the dashboard, providing real-time visibility into expert locations.
/* c/telemetry.h:111 */
static void tiers_emit(Model *m){
// ...
printf("TIERS %d %d %d %.2f %.2f\n", vram, ram, disk, vram_gb, ram_gb);
}
Sources: c/telemetry.h:111-125
Memory Hierarchy and Tiering enables Colibri to bridge the gap between model size and consumer hardware capacity. By treating weights as stageable data rather than fixed state, the system achieves real-time inference speeds even when the majority of the model resides on NVMe storage.