Unit Testing & CI Framework

Relevant source files

The following files were used as context for generating this wiki page:

• CONTRIBUTING.md
• c/tools/diag_harness.py
• c/tests/test_efficiency_report.py
• c/tests/README_efficiency.md
• c/tests/test_schema_gbnf.c
• README.md

The Colibrì testing and Continuous Integration (CI) framework ensures the reliability, correctness, and efficiency of the inference engine. It consists of multiple layers of validation, ranging from portable CPU-only unit tests to deep diagnostic harnesses that verify token-exact parity with reference models.

The framework prioritizes a dependency-free default CPU path while providing specialized tests for CUDA backends and performance optimization. It uses a combination of C-based unit tests for core logic and Python-based harnesses for high-level model validation and throughput measurement.
Sources: CONTRIBUTING.md:1-20, c/tests/README_efficiency.md:1-10

Testing Architecture

The testing system operates across several distinct levels of complexity and scope.

Validation Layers

Layer	Component	Description
Unit Tests	make check	Portable C unit tests and Python standard-library tests. No model required.
Regression	test_inefficiency.py	Fast asserted tests for throughput and accounting sanity using glm_tiny.
Correctness	diag_harness.py	Validates model responses against expected outputs and detects repetition loops.
Parity	Oracle Check	Token-exact validation against transformers reference (teacher-forcing 32/32).
Efficiency	test_efficiency_report.py	Detailed optimization dossier for real model runs.

Sources: CONTRIBUTING.md:22-30, c/tests/README_efficiency.md:46-55, c/tools/diag_harness.py:10-25

Diagnostic Workflow

The diagnostic harness (diag_harness.py) follows a multi-phase execution flow to certify a model snapshot.

Correctness & Parity Testing

Core correctness is verified through automated smoke tests and grammar-based validation.

Model Smoke Testing

The DiagnosticHarness executes a suite of curated prompts across categories like factual knowledge, math, reasoning, and code generation. It verifies the generated text for three main criteria:

1. Presence: The engine must generate non-empty text.
2. Parity: Expected substrings must appear for factual queries.
3. Coherence: The generator must not enter degenerate repetition loops (e.g., repeating the same token 3+ times).

Sources: c/tools/diag_harness.py:34-60, c/tools/diag_harness.py:255-285

Grammar & Schema Validation

The test_schema_gbnf.c file provides end-to-end testing for the JSON-Schema to GBNF (Guidance BNF) compiler. It ensures that compiled grammars correctly drive the Pushdown Automaton (PDA).

• Key Functions: gr_parse to load GBNF, gr_accept to walk byte strings, and gr_forced to identify spans that the grammar mandates.
• Test Cases: Strict objects, enums, nested arrays, and escaped characters.

Sources: c/tests/test_schema_gbnf.c:11-100

Efficiency & Performance Dossier

Performance optimization is handled by the test_efficiency_report.py script, which generates a comprehensive optimization dossier.

Instrumentation Flags

To generate a report, the engine is run with maximum observability toggles enabled:

• PROF=1: Enables high-level profiling of wall-time shares.
• LOOKA=1: Enables router-lookahead telemetry.
• DISK_SPLIT=1: Details I/O activity across decode phases (draft vs. verify).
• CACHE_ROUTE=1: Measures routing agreement and KL divergence.

Sources: c/tests/test_efficiency_report.py:46-60, c/tests/README_efficiency.md:12-20

Efficiency Metrics & Thresholds

The framework defines advisory thresholds to identify bottlenecks.

Metric	Threshold	Indicated Issue
Disk-Wait Share	> 40%	I/O-bound (Slow disk or small cache)
Cache Hit Rate	< 30%	Cache thrashing
Route Agreement	< 80%	Prefetcher misprediction
MTP Acceptance	< 20%	Draft head is dead weight
Tail Ratio (p99/p50)	> 3.0x	Stalls due to I/O hiccups or KV growth

Sources: c/tests/test_efficiency_report.py:34-40, c/tests/README_efficiency.md:27-35

Continuous Integration Process

Colibrì uses a two-branch strategy to maintain stability.

1. main Branch: The stable branch where the engine must always pass the token-exact oracle.
2. dev Branch: The integration branch for new Pull Requests.

PR Requirements

Every Pull Request is reviewed against the following criteria:

• Clean Build: Zero compiler warnings.
• Oracle Parity: 32/32 teacher-forcing (TF) agreement + 20/20 greedy agreement.
• Local Checks: Running make check to verify C unit tests and Python standard-library tests.
• CUDA Validation: CUDA-specific changes must be checked on capable hosts using make -C c cuda-test.

Sources: CONTRIBUTING.md:5-20, CONTRIBUTING.md:32-37

Summary

The Colibrì Unit Testing & CI framework provides a structured approach to maintaining a high-performance inference engine. By combining fast regression tests for CI gating with deep diagnostic and efficiency reports for architectural optimization, the project ensures that the 744B parameter model remains faithful to the reference implementation across diverse hardware configurations.
Sources: README.md:220-250, c/tests/README_efficiency.md:37-44