Performance Benchmarks
Relevant source files
The following files were used as context for generating this wiki page:
Performance Benchmarks in Colibri provide a multi-layered framework for measuring the inference speed, hardware efficiency, and output quality of large Mixture-of-Experts (MoE) models like GLM-5.2. The system evaluates how weights are staged across the memory hierarchy, including VRAM, RAM, and NVMe storage. These benchmarks help developers identify bottlenecks such as I/O stalls, cache thrashing, or compute limitations on various hardware configurations.
The benchmarking suite consists of diagnostic harnesses for automated testing, efficiency reports for deep architectural profiling, and user-experience scripts for real-world latency measurement. By streaming experts from disk, the project enables high-parameter models to run on consumer hardware while providing technical metrics to verify that speed optimizations do not cause output drift.
Sources: README.md:14-22, c/tests/README_efficiency.md:3-9, site/index.html:158-164
Diagnostic Harness Architecture
The diag_harness.py tool executes a structured campaign of tests to validate model snapshots. It captures engine telemetry by parsing stdout and stderr streams against specific format strings defined in the C engine. The harness operates in five distinct phases to provide a complete view of system health and performance.
Figure 1: The execution flow of the Diagnostic Harness showing the sequence of testing phases.
Test Phases
- System Probe: Captures startup telemetry including load time, resident memory, layer/expert counts, and CUDA device information.
- Smoke Test: Validates correctness using curated prompts to check for coherence and severe token repetition.
- Deep Diagnostic: Extracts full
PROFILEbreakdowns, measuring time spent in expert-disk I/O, matmul, and attention. - Quality Benchmarks: Runs accuracy tests via
eval_glm.pyon datasets like HellaSwag and MMLU. - Throughput: Measures tokens per second (tok/s) with Mixture-of-Experts (MTP) speculation enabled and disabled to compute speedup.
Sources: c/tools/diag_harness.py:8-17, c/tools/diag_harness.py:180-260
Efficiency and Optimization Dossier
The efficiency suite provides an optimization dossier that identifies subsystems consuming decode wall-time. It uses advisory thresholds to flag inefficiencies, such as when disk I/O wait dominates the decode process.
Advisory Thresholds
The system uses the following defaults to flag potential bottlenecks:
| Threshold Constant | Value | Description |
|---|---|---|
DISK_WAIT_DOMINANT | 0.40 | >40% decode waiting on expert reads (I/O-bound) |
LOW_HIT_RATE | 0.30 | <30% cache hit (thrashing/cap too small) |
LOW_ROUTE_AGREE | 0.80 | <80% routing overlap (prefetch guessing wrong) |
HIGH_TAIL_RATIO | 3.0 | p99 > 3x p50 (decode stalls/KV growth) |
LOW_MTP_ACCEPT | 0.20 | <20% MTP acceptance (draft decoder is dead weight) |
Sources: c/tests/test_efficiency_report.py:31-36, c/tests/README_efficiency.md:36-45
Hardware Staging and Memory Tiers
The performance is heavily influenced by how experts are distributed across the storage hierarchy. The engine measures "routing heat" to decide placement.
Figure 2: The tiered storage architecture for weight placement and streaming.
Sources: README.md:69-80, site/index.html:158-164, c/telemetry.h:115-125
Performance Metrics and Telemetry
The engine emits telemetry lines that the benchmarking tools parse to generate reports. Key metrics include throughput, prefill latency, and expert hit rates.
Key Performance Identifiers
- tok/s: The primary throughput metric for decoding.
- TTFT (Time To First Token): Prefill wall-time measured in seconds.
- Expert Hit Rate: Percentage of expert requests satisfied by the internal cache (VRAM/RAM) without disk access.
- MTP Acceptance: The percentage of tokens generated by the speculative draft head that are verified by the main model.
# Regex patterns for metric extraction from engine output
RX = {
"decode_tps": (re.compile(r"decode \d+ tokens in [\d.]+s \(([\d.]+) tok/s\)"), lambda m: float(m.group(1))),
"hit_rate": (re.compile(r"expert hit rate ([\d.]+)%"), lambda m: float(m.group(1))),
"mtp_accept": (re.compile(r"MTP acceptance ([\d.]+)%"), lambda m: float(m.group(1))),
"prof_expert_disk": (re.compile(r"expert-disk ([\d.]+)s service"), lambda m: float(m.group(1))),
}
Sources: c/tools/diag_harness.py:80-100, c/tools/bench_ux.sh:39-42
User Experience (UX) Benchmarking
The bench_ux.sh script focuses on real-world scenarios rather than synthetic micro-benchmarks. It runs fixed scenarios (e.g., short chat vs. long document analysis) and reports medians across multiple repetitions to account for variance in .coli_usage and disk I/O.
Scenario Definitions
- Chat Scenario: Uses a short prompt ("Explain why the sky is blue") to measure low-context latency.
- Longdoc Scenario: Uses a large Chinese text block repeated 12 times to test prefill performance and long-context handling.
Sources: c/tools/bench_ux.sh:22-35
Conclusion
Performance Benchmarks in Colibri serve as the primary tool for validating the efficiency of the three-tier memory hierarchy. By monitoring metrics like expert hit rates and disk-wait shares, developers can tune environment variables such as RAM_GB, PIN_GB, and CUDA_EXPERT_GB to maximize tokens per second on specific hardware. The suite ensures that the MoE engine remains precise and fast, even when operating under significant memory constraints.
Sources: c/tests/test_efficiency_report.py:270-280, c/tests/README_efficiency.md:25-34