Extending Hardware Backends

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• SUMMARY.md
• CHANGELOG.md
• c/telemetry.h
• c/tests/test_efficiency_report.py

The colibri engine employs a modular hardware abstraction layer to support various compute environments, ranging from CPU-only consumer laptops to multi-GPU clusters. It treats VRAM, RAM, and storage as a single managed memory hierarchy, allowing Mixture-of-Experts (MoE) models like GLM-5.2 to run on diverse hardware by adjusting where specific parameters reside.

The architecture separates the core engine logic from backend-specific implementations, such as CUDA for NVIDIA GPUs, HIP/ROCm for AMD GPUs, and Metal for Apple Silicon. This enables developers to introduce new backends or optimize existing ones by implementing standardized interfaces for memory allocation, tensor matmul, and expert staging.

Sources: README.md:16-24, CHANGELOG.md:144-150

Multi-Tiered Storage Hierarchy

Colibri stages model weights across three primary hardware tiers. The placement policy is deterministic: the dense part of the model remains resident in fast memory (VRAM or RAM), while routed experts are streamed from slower tiers based on demand and measured routing heat.

Storage Tiers

Tier	Hardware	Role
Hot	VRAM	Stores resident dense weights and the "hottest" experts for low-latency GPU execution.
Warm	RAM	Holds dense weights on CPU-only runs and acts as a pinned hot-store for frequently used experts.
Cold	NVMe / Disk	Acts as the primary storage for the 19,456 experts, streamed on-demand.

Sources: README.md:92-100, CHANGELOG.md:146-148

Backend Implementation Architecture

Hardware backends are integrated via specific source modules. For example, backend_cuda.c handles NVIDIA-specific logic, while backend_metal.mm supports Apple Silicon. A single-source backend_gpu_compat.h provides a WMMA dispatch gate, allowing a unified codebase for both CUDA and HIP (AMD) support.

Key Backend Functions

The engine requires backends to handle specific lifecycle and execution tasks:

• Initialization: coli_metal_init or coli_cuda_init sets up pipeline descriptors and command queues.
• Registration: coli_metal_register wraps malloc'd host memory into backend-specific buffers (e.g., MTLBuffer).
• Submission: moe_submit dispatches kernels (like moe_gemv) to execute Expert matmuls on the hardware.
• Telemetry: Backends must report hardware-specific stats, such as VRAM usage and kernel execution times.

Sources: CHANGELOG.md:88-93, SUMMARY.md:15-30, c/telemetry.h:87-105

Hardware Probing

The hw_probe function identifies system capabilities at startup to inform the placement engine. It detects CPU models, core counts, and total system RAM across Windows, macOS, and Linux.

Performance Optimization Levers

When extending backends, specific environment variables and flags control how the hardware is utilized. These knobs allow for performance tuning based on the specific bottleneck of the target hardware.

Configuration Knobs

Variable	Function	Recommended Use
COLI_NUMA	Interleaves weights across memory controllers.	Essential for multi-socket CPU hosts.
COLI_CUDA_PIPE	Keeps residual stream on-device across layers.	Reduces CPU-GPU sync overhead (Mode 2).
PIPE	Enables async I/O pool for loading experts.	Overlaps compute with NVMe reads.
PILOT	Prefetches experts for the next layer.	Hides latency based on routing predictability.
DIRECT	Bypasses page cache for NVMe reads.	Optimizes throughput on high-speed SSDs.

Sources: README.md:120-135, CHANGELOG.md:156-159, c/tests/test_efficiency_report.py:76-85

Telemetry and Diagnostics

Extended backends must integrate with the telemetry.h protocol to provide live visibility into the hardware's performance. The engine uses this data to update the "Brain" and "Atlas" visualizations in the web dashboard.

• Tiers Emission: tiers_emit calculates the distribution of experts across VRAM, RAM, and Disk to update the UI tier bar.
• Efficiency Dossier: The test_efficiency_report.py script exercises instrumentation flags like PROF, DISK_SPLIT, and LOOKA to generate optimization reports.
• Expert Mapping: emap_emit generates a hex-encoded bitmap of expert usage and residency tiers for the live cortex view.

Sources: c/telemetry.h:118-135, c/telemetry.h:137-160, c/tests/test_efficiency_report.py:1-15

Conclusion

Extending hardware backends in colibri involves implementing specialized memory management and kernel dispatch logic while adhering to the three-tier residency model. By leveraging unified headers for GPU compatibility and providing detailed telemetry, new hardware can achieve token-exact results while optimizing for its specific architectural strengths.

Sources: README.md:10-15, CHANGELOG.md:88-93