Runpod Deployment Guide
Relevant source files
The following files were used as context for generating this wiki page:
This guide describes the deployment and execution architecture of the Runpod serverless runner for Ollama. The system is designed to wrap the Ollama inference engine within a Runpod serverless environment, providing an OpenAI-compatible interface for LLM operations. The runner supports both standard prompt-based generation and chat completions with full streaming capabilities.
Sources: README.md:1-13, src/handler.py:7-34
Core Architecture and Execution Flow
The deployment begins with an entrypoint script that initializes the Ollama environment before starting the Python-based worker. The system utilizes a dual-engine approach: one for standard LLM inputs and another specifically for OpenAI-formatted requests.
Startup Sequence
The initialization process is managed by src/start.sh, which performs the following steps:
- Cleans up existing Ollama processes.
- Starts the Ollama server in the background and waits for it to become available by monitoring logs for the "Listening" state.
- Optionally pulls a model defined by the
OLLAMA_MODEL_NAMEenvironment variable. - Launches the Runpod handler.
Sources: src/start.sh:3-48
Request Handling Flow
When a job is received, the handler function determines the appropriate engine based on the presence of an OpenAI route in the input.
The flow shows how the worker discriminates between standard and OpenAI-compatible requests to route data to the correct processing engine.
Sources: src/handler.py:10-22, src/utils.py:1-6
Configuration and Environment
The worker's behavior is primarily controlled through environment variables and build-time arguments.
Environment Variables
| Variable Name | Description | Default Value |
|---|---|---|
OLLAMA_MODEL_NAME | The name of the model to download/use during runtime. | NULL (or llama3.2:1b in code) |
MAX_CONCURRENCY | Maximum number of concurrent jobs the worker handles. | 8 |
Sources: README.md:15-19, src/handler.py:6-7, src/engine.py:22
Preloading Models
To avoid latency during cold starts, models can be embedded directly into the Docker image using build arguments.
docker build --build-arg MODEL_NAMES="<model-name>,..." -t <yourname>/<yourrepo>:<tag> .
Sources: embed_model/README.md:3
The preload_model.sh script automates this by starting a temporary Ollama server during the build process to pull the specified models.
Sources: embed_model/preload_model.sh:3-33
Engine Logic and API Compatibility
The project implements two primary engine classes in src/engine.py to facilitate interaction with the Ollama backend.
OllamaOpenAiEngine
This engine maps Runpod job inputs to the local Ollama OpenAI-compatible API endpoint (http://localhost:11434/v1/). It handles three main routes:
/v1/models: Returns a list of available models./v1/chat/completions: Handles structured chat interactions./v1/completions: Handles raw prompt completions.
Sources: src/engine.py:7-14, src/engine.py:58-75
Data Models
Input data is structured using the JobInput class, which extracts key fields from the Runpod job payload.
class JobInput:
def __init__(self, job):
self.llm_input = job.get("messages", job.get("prompt"))
self.stream = job.get("stream", False)
self.openai_route = job.get("openai_route")
self.openai_input = job.get("openai_input")
Sources: src/utils.py:1-6
Streaming Implementation
For streaming requests, the engine yields data chunks in an OpenAI-compatible format, ending the stream with a [DONE] marker. Non-streaming requests yield the complete response as a dictionary.
Sources: src/engine.py:90-106
Summary
The Runpod Ollama worker provides a bridge between Runpod's serverless infrastructure and the Ollama inference engine. By utilizing src/start.sh for lifecycle management and src/engine.py for API abstraction, it allows for scalable LLM deployment with minimal configuration, supporting both direct prompts and standard OpenAI integrations.