Resource Planner & Diagnostics

Relevant source files

The following files were used as context for generating this wiki page:

• c/doctor.py
• c/tools/diag_harness.py
• c/tests/test_doctor.py
• c/tests/test_efficiency_report.py
• README.md

The Resource Planner & Diagnostics system provides tools to evaluate system readiness, plan memory allocation, and verify model integrity for the colibri engine. It ensures that the large-scale GLM-5.2 Mixture-of-Experts (MoE) model can run efficiently by managing a memory hierarchy consisting of VRAM, RAM, and NVMe storage.

This system consists of the doctor utility for pre-flight readiness checks, a diagnostic harness for multi-phase performance testing, and efficiency reporting tools that identify bottlenecks in the expert-streaming pipeline. These components allow users to run 744B-parameter models on consumer hardware by validating that the environment meets specific hardware and software constraints.

Sources: README.md:16-24, c/doctor.py:4-10

System Diagnostics (Doctor)

The doctor module performs read-only installation diagnostics to verify the environment before model execution. It checks the presence and validity of core model files, hardware availability, and engine binary integrity.

Diagnostic Checks

The run_doctor function executes several validation categories:

• Model Integrity: Verifies config.json, tokenizer.json, and safetensors shards.
• Hardware Detection: Discovers GPUs and measures available system RAM.
• Engine Readiness: Checks if the colibri binary exists, is executable, and has all required shared libraries (e.g., libgomp.so.1).
• Storage Access: Ensures the model directory is readable and checks if it has write permissions for persistence files like .coli_usage and .coli_kv.

Sources: c/doctor.py:246-320, c/doctor.py:228-243

Deep Container Validation

When the --deep flag is used, the system performs a detailed byte-level inspection of the model container. This includes:

1. Header Parsing: Validates safetensors headers for JSON correctness and length.
2. Tensor Layout: Checks for overlapping tensor ranges and ensures shapes match declared dtypes.
3. Shard Sequencing: Verifies that shard numbering (e.g., out-000.safetensors) is contiguous and complete.
4. Index Verification: Matches the model.safetensors.index.json weight map against actual discovered tensors.

Sources: c/doctor.py:126-200, c/tests/test_doctor.py:129-150

Diagnostic Harness

The diag_harness.py utility runs a multi-phase test campaign against model snapshots. It automates telemetry capture, correctness testing, and performance benchmarking.

Test Phases

The harness organizes testing into five distinct phases:

Phase	Name	Description
0	System	Probes startup telemetry (load time, GPU VRAM, RAM budget, idot kernel).
1	Smoke	Checks correctness using curated prompts for factual, math, and coding tasks.
2	Diagnostic	Performs deep profiling of timing breakdowns (expert-disk vs. expert-matmul).
3	Quality	Runs accuracy benchmarks such as HellaSwag, MMLU, and ARC-Challenge.
4	Throughput	Measures tokens per second with and without MTP speculation.

Sources: c/tools/diag_harness.py:8-18, c/tools/diag_harness.py:126-160

Performance Metrics Extraction

The harness uses regex to extract metrics from the engine's stdout and stderr. Key metrics include:

• Expert Hit Rate: The percentage of expert requests served from the RAM/VRAM cache.
• MTP Acceptance: The percentage of tokens drafted by the Multi-Token Prediction head that were verified by the main model.
• Time Shares: The share of total decode time consumed by attention, expert-disk I/O, and matmul operations.

Sources: c/tools/diag_harness.py:46-88, c/tests/test_efficiency_report.py:125-140

Efficiency Reporting and Optimization

The efficiency system identifies bottlenecks in the MoE execution pipeline. It provides a dossier that evaluates whether the engine is I/O-bound or compute-bound.

Resource Bottleneck Indicators

The system uses advisory thresholds to flag inefficiencies:

• I/O Bound: If disk-wait share exceeds 40% of decode time.
• Thrashing: If the expert cache hit rate falls below 30% (indicating the RAM budget or cache cap is too small).
• Routing Drift: If route agreement between predicted and true experts is below 80%.
• Speculation Overhead: If MTP acceptance is below 20%, suggesting the draft decoder is a net performance cost.

Sources: c/tests/test_efficiency_report.py:32-38, c/tests/test_efficiency_report.py:180-210

Hardware Hierarchies

Resource planning differentiates between residency tiers. Dense weights (attention, shared experts) are resident in RAM, while routed experts are tiered across VRAM, RAM, and NVMe based on measured "routing heat" from the .coli_usage file.

Sources: README.md:92-108, c/doctor.py:300-315

Conclusion

The Resource Planner & Diagnostics system is essential for maintaining model quality and performance across diverse hardware. By providing granular timing breakdowns and model integrity checks, it enables developers to optimize the tiered storage hierarchy for 700B-class models on local machines.