Tool Calling & Structured Outputs
Relevant source files
The following files were used as context for generating this wiki page:
Tool Calling and Structured Outputs allow the Colibri engine to interact with external functions and produce data in strictly defined formats. This system translates OpenAI and Anthropic API requests into model-specific prompts and parses the resulting text back into structured objects. It supports native GLM-5.2 XML-based tool calling and grammar-forced speculative drafting for JSON outputs.
Sources: c/openai_server.py:1-20, README.md:143-157
Architecture and Data Flow
The openai_server.py gateway acts as a translation layer between HTTP clients and the colibri C engine. It handles chat template rendering, tool call extraction, and grammar enforcement.
The diagram shows the end-to-end flow from an API request to the engine and back through the structured parsers.
Sources: c/openai_server.py:640-700, c/tests/test_openai_tools_e2e.py:24-60
Tool Calling Implementation
The engine supports function calling by wrapping declarations in XML-like tags. The GLM-5.2 model identifies these tags to signal intent to use a tool.
Prompt Rendering
When tools are provided in a request, the gateway prepends a system block containing function signatures within <tools> tags. It instructs the model to use a specific XML format for calls.
Sources: c/openai_server.py:348-370
Response Parsing
The gateway parses assistant responses for <tool_call> blocks. It extracts the function name and maps <arg_key> and <arg_value> pairs into a JSON arguments object.
| Feature | Description | Tag/Marker |
|---|---|---|
| Tool Call Start | Signals the beginning of a tool request. | <tool_call> |
| Tool Call End | Signals the end of a tool request. | </tool_call> |
| Argument Key | Wraps the parameter name. | <arg_key> |
| Argument Value | Wraps the parameter value. | <arg_value> |
| Observation | Signals the start of tool results. | `< |
| Response Wrap | Wraps the result content. | <tool_response> |
Sources: c/openai_server.py:136-146, c/openai_server.py:372-388
Recovery and De-mangling
The gateway includes logic to handle common failures in quantized model outputs:
- Unclosed Tags: If a budget limit prevents the model from emitting
</tool_call>, the parser attempts to recover the partial tail if it is unambiguous. - Salvage Mode: Enabled via
COLI_TOOL_SALVAGE=1, this mode maps unstructured payloads to a tool's primary parameter if the model fails to emit<arg_key>tags.
Sources: c/openai_server.py:175-194, c/openai_server.py:228-245
Structured Outputs with GBNF Grammars
Colibri uses GBNF (Guidance Backus-Naur Form) grammars to force outputs into specific structures, such as JSON.
Grammar-Forced Drafts
Unlike traditional sampling constraints, Colibri applies grammars during the speculative drafting phase. The MTP (Multi-Token Prediction) head generates drafts that must comply with the grammar before the main model verifies them. This provides "free" structured output without slowing down the primary forward pass.
Sources: README.md:150-157, CHANGELOG.md:144-147
Supported Formats
The response_format parameter in the API supports three structured modes:
| Type | GBNF Source | Implementation |
|---|---|---|
json_object | GENERIC_JSON_GBNF | A pre-defined, whitespace-tolerant JSON grammar. |
json_schema | Dynamic | Compiles a provided JSON Schema into GBNF via schema_gbnf.h. |
gbnf | User-provided | Accepts raw GBNF strings for custom constraints. |
Sources: c/openai_server.py:535-565, c/openai_server.py:590-610
Anthropic Messages API Translation
The project includes a translation layer for the Anthropic Messages API (/v1/messages). This allows tools like Claude Code to work with the Colibri engine without modification.
The sequence diagram illustrates how Anthropic-specific tool blocks are mapped to the internal GLM format.
Sources: c/openai_server.py:730-760, CHANGELOG.md:32-41
Diagnostic and Performance Metrics
Structured output and tool calling performance can be monitored via the diag_harness.py and the web dashboard.
- Acceptance Rate: MTP draft acceptance is tracked; low rates (<20%) suggest the grammar or draft head is not helping throughput.
- Expert Mapping: The
experts.jsonatlas identifies specialized experts that fire for specific topics likejson_formatorcode_python.
Sources: c/tools/diag_harness.py:75-90, c/tools/expert_atlas/probes.json:50-60
Summary
The Tool Calling and Structured Output system provides robust integration for external applications by combining strict XML parsing for functions and GBNF-guided speculation for data formats. Features like COLI_TOOL_SALVAGE and unclosed tag recovery ensure reliability even when running heavily quantized models on consumer hardware.