OpenAI-Compatible API

Relevant source files

The following files were used as context for generating this wiki page:

• c/openai_server.py
• README.md
• CHANGELOG.md
• c/tools/diag_harness.py
• web/src/lib/api.test.ts
• web/README.md

The colibrì OpenAI-Compatible API provides a standardized HTTP gateway for interacting with the underlying C inference engine. It acts as a bridge between the stateless OpenAI API specification and the stateful, tiered memory architecture of the colibrì engine. This module enables integration with standard tools like Claude Code, the Anthropic SDK, and the built-in web dashboard.

Sources: c/openai_server.py:1-20, README.md:213-220, CHANGELOG.md:46-63

Architecture and Data Flow

The API system consists of a Python-based HTTP server that manages an instance of the colibrì C engine. It handles request scheduling, prompt rendering, and response streaming.

System Components

The gateway manages concurrent requests using a bounded FIFO admission queue and maps them to independent KV contexts (slots) in the engine.
Sources: c/openai_server.py:73-85, c/openai_server.py:710-725, web/README.md:1-15

Generation Flow

When a request arrives, the server validates the input, renders a model-specific chat template, and admits the request to the scheduler. For streaming requests, the server utilizes a ThinkingStreamSplit or InklingStreamSplit to handle model-specific reasoning markers before emitting Server-Sent Events (SSE).

API Endpoints

The server exposes standard OpenAI endpoints and colibrì-specific diagnostic routes.

Endpoint	Method	Description
/v1/chat/completions	POST	Generates a chat response using OpenAI-style messaging.
/v1/completions	POST	Generates a response from a raw text prompt.
/v1/messages	POST	Anthropic-compatible Messages API for Claude Code support.
/v1/models	GET	Returns a list containing the currently loaded model.
/health	GET	Returns engine status, scheduler metrics, and hardware info.
/profile	GET	Returns a rolling window of per-turn timing snapshots.
/experts	GET	Returns a live map of expert routing and VRAM/RAM/Disk hits.

Sources: c/openai_server.py:1080-1150, CHANGELOG.md:46-63, web/README.md:15-25

Core Classes and Logic

GenerationScheduler

Manages access to the engine's KV context slots. It prevents head-of-line blocking by allowing requests targeting free slots to bypass waiters pinned to busy slots.

• Key Fields: free_slots, active, queue, capacity.
• Method admit: Uses a threading.Condition to block until a suitable slot is available or a timeout occurs.
  Sources: c/openai_server.py:73-138

Engine Wrapper

Manages the colibri C subprocess via stdin and stdout.

• SUBMIT: Writes a binary header and prompt to the engine.
• _dispatch_stdout: A background thread that parses engine frames: DATA, ACCEPT, DONE, HWINFO, EMAP, HITS, PROF, TIERS, and ERROR.
  Sources: c/openai_server.py:650-715

Chat Rendering

Converts JSON message arrays into raw text prompts based on the model architecture (ARCH).

• GLM-5.2: Uses <|user|>, <|assistant|>, and <think> markers.
• Inkling: Uses <|message_user|>, <|message_model|>, and <|content_text|> markers.
  Sources: c/openai_server.py:350-480

Configuration Options

Options can be set via command-line arguments or environment variables.

Parameter	Env Variable	Default	Description
--port	-	8000	The HTTP port to listen on.
--max-tokens	NGEN	1024	The global generation token cap.
--kv-slots	COLI_KV_SLOTS	1	Number of independent KV contexts.
--max-queue	COLI_MAX_QUEUE	8	Maximum pending requests in the scheduler.
--api-key	COLI_API_KEY	None	Bearer token required for authentication.
--allowed-host	COLI_ALLOWED_HOSTS	loopback	Hostnames trusted by the DNS-rebinding guard.

Sources: c/openai_server.py:1220-1250, c/tools/diag_harness.py:270-285

Multi-Turn KV Caching

Colibrì improves performance by persisting conversation state.

1. cache_slot: Clients can manually specify a slot ID or the server calculates one by hashing the initial conversation messages.
2. Stateless to Stateful Mapping: Hashing stable prefix messages ensures that subsequent turns of the same conversation land on the same engine slot, enabling prefix reuse.
3. Prefix Matching: The C engine detects the cached prefix to avoid redundant "prefill" computation.
   Sources: c/openai_server.py:530-555, web/src/lib/api.test.ts:60-75

Anthropic Translation Layer

The /v1/messages endpoint translates Anthropic-shaped requests into internal OpenAI-style requests.

• Content Blocks: Maps text, tool_use, and tool_result blocks to corresponding OpenAI roles.
• Reasoning: Maps Anthropic thinking blocks to reasoning_content.
• Stop Reasons: Maps end_turn to stop and max_tokens to length.
  Sources: c/openai_server.py:590-645, CHANGELOG.md:46-63