Introduction and Vision
Relevant source files
The following files were used as context for generating this wiki page:
Colibrì is a lightweight, pure C inference engine designed to run massive Mixture-of-Experts (MoE) models, specifically GLM-5.2 (744B parameters), on consumer-grade hardware. It operates without external dependencies or mandatory GPU clusters by treating VRAM, RAM, and disk storage as a single managed memory hierarchy. The engine prioritizes transparency and accessibility, allowing users to run, study, and improve frontier-class models locally.
The vision behind Colibrì is to remove frontier models from sealed datacenters and return them to the control of individual researchers and developers. By enabling 700B-class models to run on machines with as little as 25 GB of RAM, the project eliminates the need for expensive API subscriptions or massive cloud infrastructure. The core engine is contained within a single C file (colibri.c), emphasizing a "tiny engine, immense model" philosophy where complexity is minimized to facilitate community-driven optimization.
Sources: README.md:14-23, site/index.html:128-132, CHANGELOG.md:162-167
Core Architecture and Tiered Storage
Colibrì manages weights as data to be staged rather than static state to be held resident in memory. While a 744B MoE model is immense, it only activates approximately 40B parameters (roughly 11 GB of routed experts) per token. The engine exploits this sparsity by placing different model components across a tiered storage hierarchy based on their activation frequency and hardware availability.
Tiered Residency Logic
The engine divides the model into two primary segments:
- Dense Weights: Includes attention mechanisms, shared experts, and embeddings (~17B parameters). These stay resident in system RAM at int4 precision (~9.9 GB).
- Sparse Experts: Includes 19,456 routed experts. These are staged across VRAM, pinned RAM, or NVMe storage.
The diagram shows how the Colibrì engine stages weights across different hardware layers based on routing decisions.
Sources: README.md:46-59, site/index.html:138-144, web/src/Brain.tsx:90-95
Key Technical Features
Colibrì implements several optimization strategies to maintain performance on hardware with limited bandwidth or memory.
Weight Staging and JIT Principles
The engine functions similarly to a Just-In-Time (JIT) compiler for weights. It monitors which experts are "hot" (frequently used) and records this in a .coli_usage sidecar file. The engine then pins these experts to faster memory tiers.
| Feature | Description |
|---|---|
| Router Lookahead | A PILOT=1 thread prefetches experts for the next layer; routing is 71.6% predictable one layer ahead. |
| Dual-SSD Mirroring | Streams from two drives simultaneously using COLI_MODEL_MIRROR to aggregate bandwidth. |
| Batch-Union | Reads unique experts once for batched positions to reduce redundant disk I/O. |
| MLA KV Compression | Compresses KV state by 57x (576 floats/token) and persists it in .coli_kv. |
| Speculative Decoding | Uses native GLM-5.2 MTP heads for drafting tokens, verified by the main model. |
Sources: README.md:65-75, README.md:92-108, CHANGELOG.md:54-56, c/tools/diag_harness.py:73-85
The Expert Atlas
The "Expert Atlas" represents a characterization of the model's internal knowledge structure. Colibrì maps experts based on their measured routing affinity across different topics such as poetry, law, and code. This atlas allows developers to visualize the "mind" of the model as a 3D galaxy where specialized experts cluster by topic.
The diagram illustrates the process of categorizing experts into specialized clusters based on routing affinity.
Sources: c/tools/expert_atlas/probes.json:3-5, web/src/Brain.tsx:41-45, site/index.html:171-174
Execution Environment and Portability
Colibrì is designed for extreme portability. The runtime engine has zero external dependencies and is written in pure C.
- Platform Support: The engine runs on Linux, macOS (Metal), and Windows 11 (MinGW).
- Hardware Acceleration: It supports AVX-512, AVX2, ARM NEON, and NVIDIA CUDA. Recent updates added AMD GPU support via HIP/ROCm.
- Docker Integration: Provides a containerized environment to isolate the engine from the host system, requiring only Docker Desktop and a fast SSD.
Diagnostic Tools
The project includes a comprehensive diagnostic harness (diag_harness.py) to validate model snapshots. This harness performs:
- Phase 0 (System): Telemetry for GPU, RAM, and kernel detection.
- Phase 1 (Smoke): Correctness checks against curated prompts.
- Phase 2 (Diagnostic): Detailed profiling of routing and disk-split performance.
- Phase 3 (Quality): Accuracy benchmarks (MMLU, HellaSwag).
Sources: c/tools/diag_harness.py:10-22, docker/README.md:27-38, CHANGELOG.md:50-52, README.md:123-132
Vision Summary
Colibrì serves as a technical "microscope" for large-scale MoE models. By optimizing for weight staging rather than residency, it enables 744B models to operate at usable speeds (up to 9 tok/s on high-end consumer hardware) while maintaining a baseline of 0.05 tok/s on extremely low-resource machines. The project aims to continue adding support for open-weight families like DeepSeek, Kimi K2, and Qwen3 MoE, ensuring frontier AI remains an open field for study and improvement.
Sources: README.md:31-40, site/index.html:152-168, README.md:188-193