HIP and ROCm Support

Relevant source files

The following files were used as context for generating this wiki page:

• GPU_BACKENDS.md
• README.md
• CHANGELOG.md
• c/tools/diag_harness.py
• SUMMARY.md
• backend_gpu_compat.h

Colibrì provides Heterogeneous-compute Interface for Portability (HIP) and Radeon Open Compute (ROCm) support to enable GPU acceleration on AMD hardware. This implementation allows a single GPU expert backend source file (c/backend_cuda.cu) to compile for both NVIDIA and AMD vendors. The system uses a shim header (c/backend_gpu_compat.h) to map CUDA runtime symbols to HIP equivalents when compiled with the hipcc compiler.

Sources: GPU_BACKENDS.md:3-9, CHANGELOG.md:46-51

Unified GPU Architecture

The project maintains a unified codebase for GPU operations. The architecture relies on shared syntax for kernels, including __global__, __shared__, __syncthreads__, and the <<<>>> execution configuration syntax. Developers must place vendor-specific differences in the compatibility header rather than using #ifdef __HIP__ guards within the main backend source.

Sources: GPU_BACKENDS.md:6-11

GPU Backend Compilation Flow

The following diagram illustrates how the build system handles different GPU vendors:

This flow ensures that the NVIDIA path remains byte-identical to pre-HIP implementations while providing a 1:1 mapping for the 14-symbol CUDA runtime surface used by the backend onto the HIP runtime.

Sources: GPU_BACKENDS.md:3-9

Build and Environment Configuration

HIP and ROCm support is an opt-in feature for Linux x86-64 platforms. The default build remains a dependency-free CPU implementation. Users enable HIP support by passing HIP=1 to the makefile.

Build Specifications

Variable	Description	Default / Example
HIP=1	Enables the HIP/ROCm backend.	Mutually exclusive with CUDA=1.
ROCM_HOME	Defines the ROCm installation path.	/opt/rocm
HIP_ARCH	Specifies the target GPU architecture.	native or gfxXXXX (e.g., gfx1201).
HIP_VISIBLE_DEVICES	Masks specific GPUs at runtime.	Ordinal index (e.g., 0).

Sources: GPU_BACKENDS.md:13-20

Runtime Controls

Colibrì uses identical environment variables for runtime configuration across both AMD and NVIDIA backends.

• COLI_CUDA=1: Activates the GPU backend.
• COLI_GPU=N: Selects a specific device by ordinal.
• CUDA_EXPERT_GB=G: Sets the VRAM budget for the expert tier.
• CUDA_RELEASE_HOST=1: Causes GPU-tier experts to drop host backing after upload to save RAM.

Sources: GPU_BACKENDS.md:22-31, c/tools/diag_harness.py:180-187

Implementation Details and Limitations

Kernel Dispatch

AMD compute currently uses portable kernels. While NVIDIA hardware utilizes a COLI_GPU_HAS_WMMA gate for Tensor Core paths, the project gates these matrix-core intrinsics to prevent execution on AMD hardware that reports high compute capability but lacks identical WMMA support. Support for rocWMMA is identified as a potential future enhancement.

Sources: GPU_BACKENDS.md:32-35, CHANGELOG.md:46-51

Numerical Accuracy

GPU float matmuls exhibit different rounding behavior compared to CPU int8-dot (IDOT) kernels. Consequently, GPU output is not token-identical to CPU output. On AMD hardware, this has been observed to lower the Medusa Tree Projection (MTP) draft acceptance rate from approximately 40% to 31%.

Sources: GPU_BACKENDS.md:47-51

Validation and Testing

Validation for ROCm support involves both automated CI and hardware-specific unit tests.

Test Coverage

The hip-test suite covers:

• Correctness for q8, q4, q2, and f32 matrix multiplications.
• Multi-device placement and statistics.
• tensor_update operations.

Sources: GPU_BACKENDS.md:37-45

Hardware Compatibility

Support has been hardware-verified on the following environment:

• GPU: AMD RX 9070 XT (gfx1201)
• Software: ROCm 7.2.4, Linux 7.0
• Performance: 0.32 tok/s at 61% expert hit rate with CUDA_RELEASE_HOST=1 using GLM-5.2.

Sources: GPU_BACKENDS.md:46

ROCm support in Colibrì enables high-parameter MoE models to leverage AMD hardware while maintaining a maintainable, single-source backend architecture. This allows users to scale residency and performance across diverse GPU environments.

Sources: GPU_BACKENDS.md:52-56, CHANGELOG.md:162-167