Apple Silicon Metal Backend

Relevant source files

The following files were used as context for generating this wiki page:

• c/backend_metal.h
• SUMMARY.md
• README.md
• c/tests/test_backend_metal.mm
• c/telemetry.h

The Apple Silicon Metal Backend provides GPU acceleration for the Colibri engine on macOS devices. It executes batched expert math and fused decode attention directly on the unified-memory GPU, eliminating the need for expensive host-to-device copies. The backend compiles its shaders at runtime using newLibraryWithSource:, which removes dependencies on offline Metal compilers or Xcode.

Sources: c/backend_metal.h:11-16, README.md:144-147

Core Architecture and Initialization

The backend operates on Apple's unified memory architecture. It reads resident weights zero-copy from RAM slabs that the CPU already occupies. The system manages GPU resources through persistent handles and residency sets to minimize overhead during inference.

Component Overview

• Zero-Copy Registration: Wraps page-aligned host allocations in MTLBuffer objects using newBufferWithBytesNoCopy.
• Residency Sets: Manages resource residency globally for the process, reducing GPU stalls during frequent expert swapping.
• Unified Memory: Shares physical RAM between the CPU and GPU, allowing raw pointers to be resolved to GPU addresses.

The diagram shows how host memory slabs are aligned and registered into the Metal residency set for kernel execution.

Sources: c/backend_metal.h:12-16, c/backend_metal.h:46-52, SUMMARY.md:20-25

Resource Management and Residency

Colibri implements an experimental residency set mechanism to optimize GPU performance on macOS 15.0 and later. This mechanism declares residency once for the process lifetime instead of per command buffer, which significantly reduces GPU stalls.

Residency Set Lifecycle

1. Initialization: The engine creates one MTLResidencySet during coli_metal_init if COLI_METAL_RESSET=1 is set.
2. Registration: coli_metal_register adds new allocations to the set.
3. Flushing: moe_submit flushes pending additions to the residency set before dispatching kernels.
4. Unregistration: coli_metal_unregister removes allocations immediately before host memory is freed to prevent use-after-free errors.

Sources: SUMMARY.md:26-45, SUMMARY.md:105-115

Key Management Functions

Function	Description	Constraints
coli_metal_register	Wraps host memory in MTLBuffer and adds to residency set.	base must be 16KB aligned.
coli_metal_unregister	Removes buffer from residency set and stashes it.	Must be called before free().
coli_metal_stats	Returns count and total bytes of wrapped tensors.	N/A
coli_metal_mem_info	Provides used and total unified memory bytes.	N/A

Sources: c/backend_metal.h:23-26, c/backend_metal.h:46-52

Kernel Operations

The backend implements specialized kernels for Mixture-of-Experts (MoE) and Attention mechanisms. These kernels support multiple quantization formats, including int8, int4, and int2.

MoE Batched Execution

The coli_metal_moe_block function executes routed-expert SwiGLU operations for a layer in a single command buffer. This batching avoids the ~150us Metal launch latency otherwise paid per expert matmul.

The sequence shows the asynchronous execution of MoE blocks allowing overlap between GPU computation and disk I/O.

Sources: c/backend_metal.h:134-142, c/backend_metal.h:155-165, c/tests/test_backend_metal.mm:52-64

Attention and Decoding

coli_metal_layer_decode performs a full decode layer operation in one command buffer. This includes:

• Input RMSNorm
• Multi-Head Latent Attention (MLA) absorption
• RoPE (Rotary Positional Embeddings)
• Shared expert and Top-K routing logic

Sources: c/backend_metal.h:74-95, c/tests/test_backend_metal.mm:106-125

Telemetry and Diagnostics

The backend provides detailed performance counters that integrate with the Colibri dashboard. These metrics track both GPU execution time and CPU-side orchestration overhead.

Instrumentation Metrics

• MoE Counts: Tracks successful GPU dispatches (ok), CPU fallbacks (fb), and total experts processed.
• Timing Breakdown: Accumulates time for setup, gpu execution, and scatter operations.
• Residency Flush: Measures time spent committing residency set changes (available in coli_metal_resset_stats).

Sources: c/backend_metal.h:122-132, c/telemetry.h:103-113, SUMMARY.md:126-135

Summary

The Apple Silicon Metal Backend enables high-performance inference of large MoE models like GLM-5.2 on consumer Mac hardware. By leveraging unified memory and residency sets, it minimizes the CPU overhead of GPU dispatch. The backend is designed for efficiency, supporting asynchronous expert execution that overlaps disk I/O with GPU-based matrix multiplication across varied quantization formats.

Sources: SUMMARY.md:20-25, README.md:144-147, c/backend_metal.h:11-16