OpenAI Compatible Endpoints
Relevant source files
The following files were used as context for generating this wiki page:
The Runpod Worker Ollama provides a compatibility layer that allows users to interact with local Ollama models using the standard OpenAI API format. This feature enables seamless integration with existing tools and libraries designed for OpenAI's ecosystem, such as the openai Python client.
The system acts as a middleware that receives OpenAI-formatted requests through Runpod's serverless infrastructure, translates them if necessary, and forwards them to a local Ollama instance running at http://localhost:11434/v1/.
Sources: src/engine.py:9-13, README.md:19-20
Architecture and Request Handling
The architecture utilizes a specialized engine class, OllamaOpenAiEngine, to proxy requests to the Ollama backend. The handler determines which engine to use based on the presence of an openai_route in the job input.
Request Flow
When a job is received, the handler function instantiates the appropriate engine. If openai_route is provided in the input, OllamaOpenAiEngine is selected; otherwise, the standard OllamaEngine is used.
The standard OllamaEngine can also wrap a generic prompt into an OpenAI-compatible request by defaulting to the /v1/completions or /v1/chat/completions routes based on the input type.
Sources: src/handler.py:11-15, src/engine.py:22-48
Supported API Endpoints
The following table details the supported OpenAI-compatible routes implemented in the system:
| Route | OpenAI Method | Description |
|---|---|---|
/v1/models | client.models.list() | Lists available models currently loaded in the Ollama instance. |
/v1/completions | client.completions.create() | Legacy completion endpoint for single-string prompts. |
/v1/chat/completions | client.chat.completions.create() | Standard chat endpoint for multi-turn message lists. |
Sources: src/engine.py:65-71, src/engine.py:84-88
Model Management (/v1/models)
The engine retrieves the list of models by calling the local Ollama API. It transforms the response into a JSON object containing a list of model definitions.
Streaming and Data Handling
The system supports real-time streaming for chat and text completions. When the stream parameter is set to true, the engine yields chunks of data formatted as Server-Sent Events (SSE).
Streaming Logic
- Request Initiation: The engine calls the OpenAI client with the provided
openai_input. - Chunk Processing: If streaming is enabled, the engine iterates over the response generator.
- SSE Formatting: Each chunk is converted to a JSON string preceded by the
data:prefix. - Termination: The stream concludes with a
data: [DONE]message.
Sources: src/engine.py:92-104, README.md:19-20
Configuration and Input Schema
The JobInput utility class parses the incoming Runpod job to extract parameters required for the OpenAI request.
| Field | Source | Description |
|---|---|---|
llm_input | messages or prompt | The primary content (string or list) for the LLM. |
stream | stream | Boolean flag to enable/disable streaming (Default: False). |
openai_route | openai_route | The specific API path to target. |
openai_input | openai_input | The actual payload containing model name, temperature, etc. |
Sources: src/utils.py:1-7, test_inputs/openai_completion.json
Implementation Detail: Engine Selection
# src/handler.py:11-14
async def handler(job: any):
job_input = JobInput(job["input"])
engine_class = OllamaOpenAiEngine if job_input.openai_route else OllamaEngine
engine = engine_class()
Sources: src/handler.py:11-14
Summary
The OpenAI Compatible Endpoints module enables the Runpod worker to act as a drop-in replacement for OpenAI services by wrapping the Ollama API. It handles routing, model listing, and both synchronous and streaming completions through the OllamaOpenAiEngine class, providing a robust interface for modern LLM applications.