Live Profiling & Metrics
Relevant source files
The following files were used as context for generating this wiki page:
Live Profiling & Metrics provides real-time observability into the Colibri engine's performance, hardware utilization, and internal Mixture-of-Experts (MoE) routing behavior. The system captures granular timing data for inference phases, monitors memory hierarchy residency across VRAM, RAM, and disk, and visualizes expert activation patterns. Developers use these metrics to identify I/O bottlenecks, tune cache capacities, and verify model correctness during live generation.
Sources: web/src/Profiling.tsx, c/telemetry.h, README.md
Performance Measurement Architecture
The engine implements a multi-stage profiling pipeline that captures timing data at the C layer and exposes it via a telemetry protocol. The Profiling.tsx component polls the /profile and /experts API endpoints to update the frontend dashboard every 1500ms to 2000ms.
The diagram shows the flow of performance data from the C engine through the telemetry protocol to the web-based visualization components.
Sources: web/src/Profiling.tsx:84-98, c/telemetry.h:116-160, web/src/Brain.tsx:49-74
Inference Timing Phases
The engine tracks five primary phases during the decode cycle to determine where computational time is spent.
| Phase | Description | Source Identifier |
|---|---|---|
| I/O Wait | Time the compute thread stalls waiting for expert weights from disk. | expert_wait_s |
| Expert Matmul | Time spent performing matrix multiplication for routed experts. | expert_matmul_s |
| Attention | Time spent on KV cache lookups and attention score calculations. | attention_s |
| LM Head | Time spent projecting hidden states to the vocabulary distribution. | lm_head_s |
| Other | Scheduling, KV bookkeeping, and general overhead. | other_s |
Sources: web/src/Profiling.tsx:6-12, c/tools/diag_harness.py:58-64, c/inkling.c:116-118
Memory & Hardware Telemetry
The system monitors the distribution of model weights across the storage hierarchy. The TIERS protocol line reports how many experts reside in each memory level.
Storage Tiers
Colibri manages experts across three distinct tiers, visualized as a residency bar in the UI:
- VRAM: Hot experts pinned for GPU execution.
- RAM: Warm experts residing in the system memory LRU cache or pinned set.
- Disk: Cold experts streamed on-demand from NVMe storage.
Sources: c/telemetry.h:126-140, web/src/App.tsx:247-259
Hardware Probing
The engine performs an initial hardware probe to detect CPU models, core counts, and available physical memory. This data is used to automatically calculate the optimal expert cache capacity (cap).
This sequence illustrates the hardware detection and reporting flow initiated at engine startup.
Sources: c/telemetry.h:47-92, web/src/App.tsx:238-245
Expert Activation (The Brain Page)
The "Brain" cortex view visualizes the activation of 19,456 experts. The engine generates an EMAP (Expert Map) that represents the entire model as a grid where each cell's properties are derived from real-time usage.
Visual Indicators
- Color: Indicates the storage tier (Green for VRAM, Blue for RAM, Gray for Disk).
- Brightness: Represents "Routing Heat," which is the log2 usage count of an expert.
- Flash: A white flash indicates an expert was routed during the most recent turn.
Sources: web/src/Brain.tsx:90-116, c/telemetry.h:142-167
Affinity Mapping
Hovering over an expert in the UI reveals its "Expert Atlas" entry, which includes:
- Topic Affinity: Measured affinity for topics like Poetry, Law, Code, or Math.
- Entropy: A measure of how specialized or general the expert is.
- Role: Categorization based on layer depth (e.g., Early layers for surface features, Late layers for planning).
Sources: web/src/Brain.tsx:141-158, c/tools/expert_atlas/probes.json
Diagnostic Tooling
Beyond live UI metrics, the project includes CLI tools for deep performance analysis.
diag_harness.py
This script runs automated test phases to verify engine health:
- System Probe: Captures startup telemetry and kernel selection.
- Performance Check: Measures tokens per second (tok/s) and Time To First Token (TTFT).
- Correctness Smoke Test: Validates model output against expected factual benchmarks.
Sources: c/tools/diag_harness.py:20-33
test_efficiency_report.py
This tool generates an "Optimization Dossier" that identifies specific inefficiencies:
- Disk-Wait Dominance: Flags if >40% of time is spent on I/O.
- Low Hit Rate: Flags if <30% of experts are found in cache, suggesting the
RAM_GBorcapis too small. - Route Agreement: Measures how accurately the prefetch/pilot thread predicts expert routing.
Sources: c/tests/test_efficiency_report.py:27-35, c/tests/test_efficiency_report.py:126-140
Summary
Live Profiling & Metrics transforms Colibri from a "black box" engine into a transparent system. By providing real-time data on inference timing, memory residency, and expert routing, it enables precise hardware tuning and performance debugging. The integration between C-based telemetry and React-based visualizations allows users to see exactly how a 744B-parameter model interacts with their local hardware.
Sources: web/src/Profiling.tsx, README.md