Quick Start Guide
Relevant source files
The following files were used as context for generating this wiki page:
Colibri is a lightweight Mixture-of-Experts (MoE) runtime that executes the GLM-5.2 744B-parameter model on consumer hardware. It manages a memory hierarchy across VRAM, RAM, and NVMe storage to stream experts on demand without requiring massive clusters or cloud APIs. Sources: README.md:16-20, site/index.html:125-130
This guide provides the necessary steps to set up the Colibri engine, acquire the required model containers, and execute inference tasks using various interfaces including CLI and Docker.
System Requirements
Colibri functions by treating system memory and storage as a unified hierarchy. Sources: README.md:89-95, docker/README.md:43-52
Hardware Specifications
| Component | Minimum Requirement | Recommended / Notes |
|---|---|---|
| RAM | 24 GB - 32 GB | 48 GB+ for better performance; 25 GB is the "proven floor". |
| Storage | 400 GB free space | Fast NVMe SSD (1-15 GB/s) is essential for expert streaming. |
| GPU | Optional (CPU-only supported) | NVIDIA (CUDA) or Apple Silicon (Metal) improves speed. |
| Network | Offline capable | Required only for initial model download (~372 GB). |
Software Prerequisites
- Python 3: Used for the launcher and API gateway. Sources: README.md:143-150
- Docker Desktop: Optional, for containerized execution. Sources: docker/README.md:58-65
- C Compiler:
gccorclangwith OpenMP support (only for building from source). Sources: README.md:155-160, setup.sh:3-10
Installation
You can obtain Colibri by downloading prebuilt binaries or compiling the source code directly. Sources: README.md:139-165, setup.sh
Option 1: Prebuilt Release
- Download the archive for your platform (Linux, macOS, or Windows) from the GitHub Releases page.
- Unpack the archive.
- Ensure Python 3 is installed to run the
colilauncher.
Option 2: Build from Source
- Clone the repository:
git clone https://github.com/JustVugg/colibri && cd colibri/c. - Run the setup script:
./setup.sh. This script verifiesgccand OpenMP, builds the engine, and runs self-tests.
The diagram shows the two primary paths for setting up the Colibri environment. Sources: README.md:139-165, setup.sh
Model Acquisition
Colibri requires a specific model container, primarily the GLM-5.2 int4 group-scaled (gs64) build. Sources: README.md:172-185, docker/README.md:82-105
Download the Model
The model is approximately 372 GB. You can use the Hugging Face CLI to download it:
python -m pip install -U huggingface_hub[cli]
hf_download mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp --local-dir .
Manual Conversion
If you possess the FP8 source weights, use the Colibri converter to generate the int4 shards:
./coli convert --model /path/to/target_dir
Sources: README.md:188-190, CHANGELOG.md:162-165
Running Inference
Inference is performed via the coli command-line utility. You must specify the model path using the COLI_MODEL environment variable or a command flag. Sources: README.md:195-202, docker/README.md:129-140
Basic CLI Commands
- Interactive Chat:
COLI_MODEL=/path/to/model ./coli chat - Resource Planning:
COLI_MODEL=/path/to/model ./coli plan - Readiness Check:
COLI_MODEL=/path/to/model ./coli doctor - Web Dashboard:
./coli web --model /path/to/model(Starts API and UI on one port).
Execution Flow
This sequence illustrates the interaction between the user, the Python launcher, and the high-performance C engine during a chat session. Sources: README.md:28-35, site/index.html:125-135
Docker Deployment
Docker provides an isolated environment for running Colibri without installing local dependencies beyond Docker itself. Sources: docker/README.md:118-125, docker/README.md:143-150
Building the Image
Navigate to the folder containing the Dockerfile and run:
docker build -t colibri-i .
Starting the Container
Mount the model directory as a volume to the container:
docker run --rm -it --name colibri-c \
-v "/path/to/your/model:/app/glm-5.2" \
-e COLI_MODEL=/app/glm-5.2 \
colibri-i ./coli chat
Sources: docker/README.md:129-150
Configuration and Tuning
Colibri behavior is controlled through environment variables. Sources: README.md:104-120, README.md:213-221
| Variable | Description | Default |
|---|---|---|
COLI_MODEL | Path to the model directory. | Required |
RAM_GB | Limits the amount of RAM the engine uses. | Auto-detected |
PIPE | Enables asynchronous expert I/O pool. | 1 (Enabled) |
PILOT | Enables router-lookahead prefetch. | 1 (Enabled) |
COLI_NUMA | Interleaves weights across memory controllers. | 0 |
COLI_MODEL_MIRROR | Path to a second copy of the model for dual-SSD streaming. | None |
Sources: README.md:104-120, CHANGELOG.md:79-82
Summary
Colibri enables the local execution of frontier-scale MoE models by optimizing weight staging across available hardware tiers. By following this guide, you can transition from installation to active inference using the GLM-5.2 model, while maintaining full control over your data and hardware resources. Sources: README.md:232-238, site/index.html:125-130