Core Engine Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• c/colibri.c
• c/openai_server.py
• README.md
• CHANGELOG.md
• web/src/Brain.tsx
• c/tools/diag_harness.py
• c/tests/test_efficiency_report.py

Colibri is a lightweight Mixture-of-Experts (MoE) inference engine written in pure C. It runs massive models like GLM-5.2 (744B parameters) on consumer hardware by treating VRAM, RAM, and disk as a managed memory hierarchy. The engine prioritizes quality-preserving inference, ensuring that model precision and router semantics remain unchanged regardless of the hardware tier.

Sources: README.md:20-30, site/index.html:120-130

The architecture functions similarly to a Just-In-Time (JIT) compiler for weights. It stages only the parameters required for the current token—roughly 40B active parameters for GLM-5.2—across storage tiers. The dense part of the model remains resident in RAM, while thousands of routed experts are streamed on demand from disk based on measured routing heat.

Sources: README.md:80-95, site/index.html:145-155

Tiered Memory Hierarchy

The Colibri engine organizes model weights into three distinct tiers based on access frequency and hardware availability. This hierarchy allows models exceeding available VRAM or RAM to execute by offloading infrequent parameters to persistent storage.

• VRAM (Hot): Stores the most frequently accessed experts.
• RAM (Warm): Holds the dense model components (attention, shared experts, embeddings) and a middle tier of experts.
• NVMe (Cold): Acts as the primary repository for the full expert set (e.g., 19,456 experts for GLM-5.2).

Weight Staging Logic

The engine uses measured routing affinity to decide which experts earn residency in faster memory tiers. The router runs a layer ahead, allowing the PILOT=1 prefetch thread to hide disk latency by loading expert data before the computation thread requires it.

Sources: README.md:100-115, site/index.html:150-160, c/openai_server.py:380-400

Token Processing Pipeline

Every token generated by the engine follows a structured five-step path. This design ensures that the mathematical precision of the model is independent of the underlying storage tier.

1. Route: The router identifies the top-K experts required for the current layer.
2. Union: The engine batches unique experts to avoid redundant reads.
3. Place: The system resolves expert locations across VRAM, RAM, or Disk.
4. Overlap: Computation on resident experts occurs simultaneously with the async I/O loading of missing experts.
5. Learn: The engine updates .coli_usage to refine future caching decisions.

Sources: README.md:120-135, c/openai_server.py:410-425

Core Components and Configuration

Component	Description	Relevant Files
Engine Core	Single C file (colibri.c or glm.c) handling the forward pass.	c/colibri.c
KV Cache	Compressed MLA state (57× smaller than standard) persisted as .coli_kv.	README.md, c/colibri.c
API Gateway	OpenAI-compatible HTTP server providing streaming and scheduling.	c/openai_server.py
Scheduler	Bounded FIFO admission for independent KV context slots.	c/openai_server.py:85-150
Expert Atlas	Topic affinity map for all experts, surfaced via /experts endpoint.	web/src/Brain.tsx, tools/analyze.py

Sources: README.md:230-245, c/openai_server.py:85-100

Speculative Decoding and MTP

Colibri implements native Multi-Token Prediction (MTP) for speculative decoding. The engine uses a smaller, faster MTP head to draft potential tokens, which are then verified by the main model in a single batched forward pass.

For effective speculation, Colibri enforces two primary rules:

• Precision: The MTP head must be int8 to maintain acceptable drafting accuracy.
• Kernel Pinning: Both the draft head and the verification model must use the same kernel family (gated by SPEC_PIN=1) to prevent floating-point drift.

Sources: README.md:175-190, CHANGELOG.md:130-140

Hardware Backends and Optimization

The engine supports multiple hardware backends through single-source compatibility layers.

• CUDA: multi-GPU expert tier and dense tensor distribution. It uses COLI_CUDA_PIPE=2 to keep the residual stream on-device.
• Metal: Optimized for Apple Silicon unified memory, performing batched expert math on the GPU.
• CPU: Leverages AVX-512, AVX2, and ARM NEON kernels. It supports COLI_NUMA=1 for multi-socket memory interleaving.

Sources: README.md:150-170, CHANGELOG.md:85-100, c/tests/test_efficiency_report.py:90-110

API and Integration Architecture

The openai_server.py gateway acts as a dependency-free translation layer between the engine and external clients. It supports the OpenAI Chat Completions API and the Anthropic Messages API.

Request Flow

1. Admission: The GenerationScheduler admits requests based on available KV slots.
2. Rendering: Messages are rendered into model-specific templates (GLM or Inkling).
3. Execution: The gateway communicates with the C engine via stdin/stdout using a frame-based protocol (SUBMIT, DATA, DONE, ERROR).
4. Filtering: The StopFilter monitors the stream for role markers and sequence terminators.
5. Splitting: For thinking models, the ThinkingStreamSplit separates reasoning deltas from visible content.

Sources: c/openai_server.py:155-250, c/openai_server.py:530-600, CHANGELOG.md:35-50

Conclusion

Colibri's architecture enables frontier-class models to run on ordinary machines by moving the bottleneck from memory capacity to I/O bandwidth. By staging weights across a three-tier hierarchy and overlapping prefetch with computation, the engine maintains high model fidelity without requiring specialized datacenter hardware.

Sources: README.md:270-280, site/index.html:115-125