Supported Models & Quantizations
Relevant source files
The following files were used as context for generating this wiki page:
Colibrì supports frontier-class Mixture-of-Experts (MoE) models by utilizing a tiered memory hierarchy that streams experts from disk to RAM or VRAM. The engine prioritizes quality preservation, ensuring that quantization and routing semantics remain identical to reference implementations regardless of the hardware tier used.
Sources: README.md:16-25, site/index.html:200-210
Supported Model Families
Colibrì targets large-scale MoE models, specifically those with sparse architectures where only a fraction of parameters activate per token. The engine is designed to be model-agnostic for any MoE with routed experts.
| Model | Parameters | Developer | Status |
|---|---|---|---|
| GLM-5.2 | 744B | Z.ai | Live |
| OLMoE | 7B | Allen AI | Live |
| Inkling | 975B | Thinking Machines | Planned |
| DeepSeek | 671B–1T | DeepSeek AI | Planned |
| Kimi K2 | 1T | Moonshot AI | Planned |
| Qwen3 MoE | Variable | Alibaba Qwen | Planned |
Sources: site/index.html:220-250, CHANGELOG.md:180-195
GLM-5.2 Architecture
The flagship supported model is GLM-5.2, a 744B parameter MoE. It activates approximately 40B parameters per token, with 11GB of routed experts changing between tokens. The model features 75 MoE layers, each containing 256 experts.
Sources: README.md:58-65, site/index.html:200-210
Inkling Reasoning MoE
Inkling is a 975B reasoning model. The conversion process for Inkling involves mapping native TML tensor names to standard formats and handling interleaved gate/up rows in fused tensors.
Sources: c/tools/convert_inkling_int4.py:10-30, site/index.html:235-240
Quantization Formats
Colibrì implements several quantization formats (referred to as fmt) to balance model size and output quality.
| Format (fmt) | Description | Bit-width | Scaling Granularity |
|---|---|---|---|
| fmt=1 | int8 | 8-bit | Per-row |
| fmt=2 | int4 | 4-bit | Per-row |
| fmt=4 | Grouped-int4 | 4-bit | Group-scaled (gs64) |
| fmt=5 | int3-g64 | 3-bit | Group-scaled (gs64) |
| fmt=6 | E8/IQ3 | 3-bit | Lattice-based |
Sources: CHANGELOG.md:120-135, c/tests/test_int3.c:5-15
Group-Scaled Quantization (gs64)
The gs64 format (used in fmt=4 and fmt=5) utilizes 64-element groups for scaling. This approach significantly reduces outlier-row errors compared to standard per-row quantization. For example, int3-g64 measures 3.3x lower outlier-row error than per-row int4 while using 25% fewer bytes.
Sources: CHANGELOG.md:128-130, c/tests/test_int3.c:100-115
This diagram shows the flow of weights through the group-scaled quantization pipeline.
Sources: c/tools/convert_inkling_int4.py:35-55, c/tests/test_int3.c:20-40
Data Staging & Placement
The engine treats weights as data to be staged rather than static state. It segments the model into dense and sparse components to optimize memory usage.
- Dense Components: Attention mechanisms, shared experts, and embeddings (~17B parameters) are kept resident in RAM.
- Sparse Components: The 19,456 routed experts are streamed from disk (NVMe) or cached in RAM/VRAM based on routing heat.
Sources: README.md:65-75, site/index.html:200-210
Memory Hierarchy Mapping
The diagram illustrates how experts move through the storage hierarchy based on routing heat and prefetching.
Sources: README.md:80-95, site/index.html:205-215
Dual-SSD Streaming
To increase bandwidth, Colibrì supports COLI_MODEL_MIRROR. This allows the engine to stream experts from two drives simultaneously. Each expert is routed to a specific drive using a deterministic hash, effectively doubling the read bandwidth on disk-bound systems.
Sources: README.md:100-115, CHANGELOG.md:124-125
Conversion Workflow
Models are typically converted from FP8 or BF16 sources into the Colibrì-specific int4 or int3 containers. The conversion process is designed to be resumable and does not require the full uncompressed model to be present on disk at once.
- Download: Shards are fetched from ModelScope or HuggingFace.
- Map: Tensors are renamed to match the engine's internal expected identifiers.
- Quantize: Weights are converted to the target format (e.g.,
int4-gs64). - Save: Tensors are written to a
.safetensorscontainer along with.qsscale metadata.
Sources: c/download_fp8.py:10-30, c/tools/convert_inkling_int4.py:140-165
Summary
Colibrì supports massive models like GLM-5.2 (744B) by specializing in sparse MoE architectures. By utilizing advanced quantization like int3-g64 and a learning memory hierarchy, it enables high-quality inference on consumer-grade hardware while maintaining token-exact parity with reference models.