Integrating External Applications
Relevant source files
The following files were used as context for generating this wiki page:
The integration of external applications with the RunPod Ollama worker is achieved through a standardized interface that bridges RunPod serverless jobs with the Ollama inference engine. The system is designed to accept both native Ollama prompts and OpenAI-compatible API requests, allowing external applications to interact with the worker using familiar LLM integration patterns.
By exposing a local OpenAI-compatible endpoint (http://localhost:11434/v1/), the worker enables applications to leverage existing OpenAI client libraries while executing models locally on RunPod's serverless infrastructure. This integration supports both standard completion and chat completion workflows, including real-time streaming capabilities for low-latency feedback in external applications.
Sources: src/engine.py:8-12, README.md:16-18
Integration Architecture
The architecture facilitates communication between the RunPod environment and the internal Ollama server. When a job is received, the system determines the appropriate engine to handle the request based on the presence of OpenAI-specific routing information.
The flow demonstrates how external requests are routed through a specialized engine that interacts with the local Ollama instance.
Sources: src/handler.py:11-18, src/engine.py:22-42
Engine Logic and Data Transformation
The system utilizes two primary classes for processing: OllamaEngine and OllamaOpenAiEngine. The OllamaEngine acts as a wrapper that transforms basic prompt inputs into structured OpenAI-compatible requests, which are then processed by the OllamaOpenAiEngine.
- OllamaEngine: Handles generic inputs. If the input is a string, it constructs a
/v1/completionsrequest. If the input is a list, it treats it as message history for/v1/chat/completions. - OllamaOpenAiEngine: Directly maps requests to the local Ollama v1 API using the
openaiPython library. It handles model listing, chat, and standard completions.
Sources: src/engine.py:26-52, src/engine.py:65-71
API Interface and Request Structures
External applications interact with the worker via the JobInput structure. The integration supports three primary interaction modes, defined by the openai_route parameter.
Supported OpenAI Routes
| Route | Functionality | Input Object Requirement |
|---|---|---|
/v1/models | Lists available models in the Ollama instance | openai_input (empty) |
/v1/completions | Standard text completion | prompt, model, stream |
/v1/chat/completions | Multi-turn chat interaction | messages, model, stream |
Sources: src/engine.py:68-73, test_inputs/openai_completion.json:1-9
Job Input Configuration
The JobInput class parses the incoming RunPod job data to extract parameters necessary for external integration.
class JobInput:
def __init__(self, job):
self.llm_input = job.get("messages", job.get("prompt"))
self.stream = job.get("stream", False)
self.openai_route = job.get("openai_route")
self.openai_input = job.get("openai_input")
Sources: src/utils.py:1-6
Real-time Streaming Integration
For applications requiring real-time updates, the worker supports the OpenAI Server-Sent Events (SSE) format. When stream is set to true, the OllamaOpenAiEngine yields chunks formatted as JSON strings prefixed with data: .
The sequence shows the streaming lifecycle where tokens are yielded as they are generated by the underlying model.
Sources: src/engine.py:100-111, README.md:16-18
Environment and Runtime Configuration
External integration behavior can be modified through environment variables defined during the RunPod deployment.
| Variable | Purpose | Default |
|---|---|---|
OLLAMA_MODEL_NAME | Specifies which model to pull and use for the engine | NULL (fallback to llama3.2:1b) |
MAX_CONCURRENCY | Sets the maximum number of concurrent jobs handled by the worker | 8 |
The startup script start.sh ensures that the local Ollama server is running and the specified model is pulled before the worker begins accepting external jobs.
Sources: README.md:9-11, src/handler.py:6-7, src/start.sh:15-37
Conclusion
Integrating external applications with the RunPod Ollama worker is streamlined through its OpenAI-compatible abstraction layer. This design allows developers to use standard LLM clients to communicate with a containerized Ollama instance, supporting both legacy prompt formats and modern chat completion schemas with full streaming support.