Speculative Decoding (MTP)
Relevant source files
The following files were used as context for generating this wiki page:
Speculative Decoding in Colibrì uses Multi-Token Prediction (MTP) to accelerate inference by drafting multiple tokens simultaneously. The system employs the native MTP heads of the GLM-5.2 model to propose candidate tokens, which the main model then verifies in a single batched forward pass. This approach typically yields 2.2 to 2.8 tokens per forward pass when effective.
Sources: README.md:143-145, README.it.md:105-107
Architecture and Components
The Speculative Decoding system consists of a drafting stage and a verification stage. The draft decoder (MTP head) proposes tokens, and the verification engine ensures these tokens match the output of the main model.
MTP Draft Heads
The drafting process relies on specific MTP heads included in the GLM-5.2 model container. For optimal performance, the project requires these heads to be in int8 format. Tests show that int4 heads collapse draft acceptance rates to near zero (0–4%).
Sources: README.md:145-147, CHANGELOG.md:162-164
Verification Logic
Verification occurs when the main model performs a batched forward pass to check the validity of drafted tokens. The engine enforces a SPEC_PIN=1 default, which ensures that both the draft and verification stages utilize the same kernel family to prevent functional divergence between the two stages.
Sources: README.md:147-150, README.it.md:109-111
Data Flow Diagram
The following diagram illustrates the lifecycle of a token through the speculative decoding process:
The engine uses the native MTP head to propose sequences, then validates them against the primary model's logic.
Sources: README.md:143-150, c/tools/diag_harness.py:108-111
Efficiency and Metrics
MTP efficiency is measured by the number of tokens accepted per forward pass and the impact on total throughput (tok/s).
Performance Metrics
The engine tracks several metrics to determine if speculation provides a net benefit:
- MTP Acceptance: The percentage of drafted tokens that pass verification.
- Tokens/Forward: The average number of tokens generated per forward pass. A value > 1.0 indicates speculation is helping.
- Forwards per Token: The ratio of forward passes to generated tokens.
Sources: c/tools/efficiency.py:86-90, c/tests/test_efficiency_report.py:195-203
Speculation Statistics Table
| Metric | Description | Threshold for "Inefficient" |
|---|---|---|
| MTP Acceptance | Percentage of drafts verified by the main model | < 20% |
| Tokens per Forward | Average tokens produced per model execution | <= 1.0 |
| Speedup | Throughput ratio of (MTP ON / MTP OFF) | < 1.05x |
Sources: c/tests/test_efficiency_report.py:32, c/tools/diag_harness.py:350-355
Configuration and Tuning
Users can control the speculative decoding behavior through environment variables and command-line flags.
DRAFT=0: Disables speculation. This is recommended when the cache temperature is high or acceptance rates are low.SPEC_PIN=1: Pins draft and verify stages to the same kernel family (default).GRAMMAR=file.gbnf: Adds grammar-forced drafts, which increases acceptance rates for structured outputs like JSON.
Sources: README.md:150-153, README.it.md:113-115
Diagnostic Logic Sequence
The diagnostic harness performs specific checks to validate MTP health during model runs:
The harness compares throughput with speculation enabled and disabled to determine the "MTP Speedup" factor.
Sources: c/tools/diag_harness.py:335-357
Technical Verification
The c/tools/efficiency.py script provides regular expressions to parse MTP performance from engine logs:
# speculation: 1.05 tokens/forward (19 forwards per 20 tokens) | MTP acceptance 44% (7/16)
SPECULATION_RE = re.compile(
r"speculation:\s+([0-9.]+)\s+tokens/forward\s+\((\d+)\s+forwards per\s+(\d+)\s+tokens\)"
r"\s*\|\s*MTP acceptance\s+([0-9.]+)%"
)
Sources: c/tools/efficiency.py:86-90
Speculative Decoding (MTP) serves as a critical optimization for GLM-5.2 inference. By leveraging dedicated prediction heads and batching verification, the engine maximizes throughput on hardware where compute or I/O latency allows for drafting overhead. The system is designed to fail gracefully; if MTP acceptance drops below useful levels, the engine can revert to standard decoding to maintain efficiency.
Sources: README.md:143-150, c/tests/test_efficiency_report.py:204-206