CUDA & Multi-GPU Support

Relevant source files

The following files were used as context for generating this wiki page:

• c/backend_cuda.h
• GPU_BACKENDS.md
• c/backend_loader.c
• c/telemetry.h
• site/index.html
• README.md

Colibrì provides an optional GPU backend to accelerate the execution of large Mixture-of-Experts (MoE) models like GLM-5.2. The system supports both NVIDIA GPUs through CUDA and AMD GPUs through the HIP/ROCm framework. This support enables multi-GPU expert tiering, dense tensor distribution, and resident pipeline execution to improve token generation throughput.

The engine maintains a three-tier memory hierarchy (VRAM, RAM, and NVMe) to manage model weights. The CUDA backend allows the most frequently used experts to reside in VRAM, significantly reducing the latency associated with streaming experts from disk or system RAM.

Architecture and Backend Integration

Colibrì uses a single-source approach for GPU backends. The core logic resides in c/backend_cuda.cu, which is compiled for either NVIDIA or AMD using a shim header (c/backend_gpu_compat.h). On Windows, the engine employs a dynamic loading strategy through c/backend_loader.c to avoid strict build-time dependencies on the CUDA toolkit.

Multi-Platform Dispatch

The engine detects the available hardware at runtime. On Windows, it attempts to load coli_cuda.dll. If the DLL is missing, the engine falls back to the CPU path. On Linux, the backend is typically linked directly during the build process.

Configuration Environment Variables

The behavior of the GPU backend is controlled via specific environment variables:

Variable	Description	Default/Behavior
COLI_CUDA=1	Enables the CUDA/GPU backend.	Required for GPU use.
COLI_GPU=N	Selects a specific GPU device ID.	Defaults to device 0.
COLI_GPUS=0,1...	Selects multiple GPUs for parallel execution.	Enables multi-GPU tier.
CUDA_EXPERT_GB=G	Sets the VRAM budget for the expert tier.	Clamped to free VRAM.
CUDA_RELEASE_HOST=1	Drops host (RAM) backing for experts once uploaded to VRAM.	Default on Multi-GPU.
COLI_CUDA_PIPE=2	Enables the resident pipeline for device-side residual stream.	Optimizes CPU-GPU overlap.
Sources: GPU_BACKENDS.md:19-27, README.md:183-186

Multi-GPU Expert Tiering

Colibrì manages experts across multiple GPUs by treating VRAM as a high-speed cache. The engine tracks "routing heat" (expert usage frequency) and automatically pins the hottest experts in VRAM.

Expert Distribution

When multiple GPUs are present, the engine distributes the expert load. The CUDA_RELEASE_HOST setting allows VRAM to act as additional pinned capacity without consuming system RAM. If a GPU fails or the CPU path requires an expert that was released from host RAM, the engine rematerializes it from disk.

Multi-GPU Performance Benchmarks

Multi-GPU configurations provide significant throughput gains. For example, a 6× RTX 5090 setup can achieve full residency for a 744B model.

Hardware	Configuration	Throughput
6× RTX 5090	Full Residency + NUMA	9.0–9.2 tok/s
6× RTX 5090	Full Residency	5.8–6.8 tok/s
1× RTX 5070 Ti	GPU-resident pipeline	1.07 tok/s
Sources: site/index.html:150-155, README.md:209-215

Core API and Data Structures

The GPU backend exposes a series of functions for tensor management and specialized kernels (e.g., MLA attention, MLP).

Opaque Tensor Handles

The host engine interacts with the GPU through the ColiCudaTensor opaque handle. The host stores the pointer but never dereferences it, ensuring ABI compatibility between different compilers (e.g., MSVC and MinGW-w64).

Key Functions

Function	Purpose
coli_cuda_init	Initializes the selected GPU devices.
coli_cuda_tensor_upload	Uploads weights and scales to VRAM.
coli_cuda_expert_group	Executes a batch of experts on the GPU.
coli_cuda_attention_absorb	Performs MLA (Multi-Head Latent Attention) absorption.
coli_cuda_pipe_sync	Synchronizes the resident pipeline.
Sources: c/backend_cuda.h:18-60, c/backend_loader.c:30-70

Resident Pipeline Execution

The COLI_CUDA_PIPE=2 mode enables a "resident pipeline." This optimization keeps the residual stream on the GPU device across model layers. This prevents the overhead of transferring data back to the CPU between every layer calculation.

Validation and Testing

The project includes a suite of tests to ensure backend correctness and performance.

1. Unit Tests: The standard suite covers matmul correctness for various formats (q8, q4, q2, f32) and multi-device placement.
2. Telemetry: The c/telemetry.h module provides functions like hwinfo_emit and tiers_emit to report GPU usage and VRAM residency to the web dashboard.
3. Efficiency Reports: The diag_harness.py tool performs deep X-rays of the CUDA tier, measuring H2D (Host-to-Device) and D2H (Device-to-Host) transfer times versus kernel execution time.

Sources: GPU_BACKENDS.md:41-47, c/telemetry.h:92-110, c/tools/diag_harness.py:100-115

Hardware Compatibility Notes

• Rounding Differences: GPU float matmuls round differently than CPU integer kernels. Consequently, GPU output is not token-identical to CPU output.
• Tensor Cores: Support for W4A16 tensor-core paths is currently NVIDIA-only. AMD HIP backends utilize portable kernels instead of rocWMMA.
  Sources: GPU_BACKENDS.md:28-32, GPU_BACKENDS.md:58-62

The CUDA and Multi-GPU support allows Colibrì to scale from a single 25GB RAM laptop to high-end multi-GPU workstations, maintaining model fidelity while maximizing available hardware throughput.