OpenAI-Compatible HTTP Server

Relevant source files

The following files were used as context for generating this wiki page:

• oxidize-server/src/AGENTS.md
• oxidize-server/src/app.rs
• PRD.md
• README.md
• oxidize-c/include/oxidize/openai.h
• oxidize-cpp/src/server/main.cpp

The OpenAI-Compatible HTTP Server provides a standardized API surface for local LLM inference. It exposes endpoints that match the OpenAI API specification, allowing users to integrate oxidize with existing tools and libraries that support OpenAI-style requests. The server supports standard text completions, chat completions, embeddings, and real-time WebSocket communication.

Sources: oxidize-server/src/AGENTS.md, README.md:104-106, PRD.md:275-285

Server Architecture

The server architecture utilizes the Axum framework in Rust to assemble routes and manage application state. It follows a layered approach where middleware handles cross-cutting concerns like authentication and rate limiting before requests reach specific endpoint handlers.

Component Breakdown

Component	Responsibility
App Router	Assembles all /v1/* routes and health endpoints.
AppState	Monolithic struct holding middleware state, model runtime, and mesh configuration.
Middleware Stack	Processes requests in order: Auth -> Limits -> Audit -> Metrics.
Runtime	Bridges the core Model trait to the HTTP layer, handling the generation loop.
Mesh Cluster	Manages master node state and aggregates runners for distributed inference.

Sources: oxidize-server/src/AGENTS.md:5-20, oxidize-server/src/AGENTS.md:32-35

Request Flow Diagram

The following diagram illustrates how a client request moves through the server layers to generate a response.

API Endpoints

The server exposes standard OpenAI endpoints and additional operational routes for health monitoring and metrics.

OpenAI Compatibility Layer

Endpoint	Method	Description
/v1/chat/completions	POST	Generates responses based on a list of chat messages.
/v1/completions	POST	Performs standard text completion for a given prompt.
/v1/embeddings	POST	Extracts vector representations for input text.
/v1/models	GET	Lists all currently loaded models.
/v1/realtime	WS	Provides real-time WebSocket communication.

Sources: oxidize-c/include/oxidize/openai.h:8-14, PRD.md:275-285, README.md:237-239

Operational Endpoints

Operational routes provide visibility into the process state and are used for deployment health checks.

• /healthz: Returns a 200 status code if the process is running.
• /livez: Confirms the process is not wedged.
• /readyz: Returns 200 once the LLM model is fully loaded and ready for inference.
• /metrics: Exposes Prometheus text-based metrics for observability.
• /openapi.json: Provides the OpenAPI 3.1 schema for the server.

Sources: oxidize-c/include/oxidize/openai.h:40-52, README.md:111-112, oxidize-server/src/AGENTS.md:23

Middleware and Security

The server implements a strict middleware stacking order to ensure security and stability.

1. Authentication: Validates API keys using x-api-key or Authorization: Bearer headers. It uses constant-time comparison to prevent timing attacks.
2. Rate Limiting: Employs a continuous batching wait queue to manage request throughput.
3. Audit & Logging: Records request/response data for auditing purposes.
4. Metrics: Collects performance data for the Prometheus metrics registry.

Sources: oxidize-server/src/AGENTS.md:14-22, oxidize-server/src/AGENTS.md:32-35

Generation and Streaming

The server supports both buffered JSON responses and streaming responses using Server-Sent Events (SSE).

Streaming Sequence

The sequence diagram below describes the interaction during a streaming chat completion request.

Cross-Platform Implementations

The project maintains parallel implementations of the HTTP server logic in multiple languages to support different deployment environments.

• Rust (Axum): The primary, full-featured implementation with mesh clustering and advanced middleware.
• C++: A minimal, dependency-free POSIX socket implementation for Llama-family inference.
• C: A dependency-free C11 port that supports operational probes like /readyz.
• Go/Python: Ports targeting feature parity with the Rust core, including OpenAI-compatible schema types and completion runtimes.

Sources: oxidize-cpp/src/server/main.cpp:1-12, oxidize-c/include/oxidize/openai.h:40-52, oxidize-golang/AGENTS.md:25-30, oxidize-python/AGENTS.md:65-70

Summary

The OpenAI-Compatible HTTP Server acts as the primary interface for productionizing oxidize models. By adhering to the OpenAI API standard, it enables seamless integration with the broader AI ecosystem while providing robust operational features like rate limiting, metrics, and health monitoring for local-first LLM workflows.