Using the Pre-built Docker Image
Relevant source files
The following files were used as context for generating this wiki page:
The RunPod Ollama worker provides a streamlined way to deploy Ollama models on RunPod serverless infrastructure using a pre-built Docker image. This system automates the process of pulling models, managing the Ollama server lifecycle, and providing an OpenAI-compatible interface for inference.
Users can leverage the image svenbrnn/runpod-ollama:latest to quickly start a worker. The worker handles both standard LLM prompts and OpenAI-style chat completions, supporting features like streaming and automatic model pre-loading.
Sources: README.md:3-5, src/engine.py:11-20
Deployment and Configuration
Deploying the worker involves specifying the Docker image in the RunPod console and configuring environment variables to control model selection and concurrency.
Key Environment Variables
| Variable Name | Description | Default Value |
|---|---|---|
OLLAMA_MODEL_NAME | The name of the model to download and use | NULL (Defaults to llama3.2:1b in engine) |
MAX_CONCURRENCY | Maximum number of concurrent jobs the worker handles | 8 |
Sources: README.md:9-12, src/handler.py:6-7, src/engine.py:24
Initialization Process
When the container starts, the start.sh script manages the initialization of the Ollama environment. It starts the Ollama server, waits for it to become ready by checking logs, and optionally pulls the specified model before launching the Python handler.
The startup sequence ensures the Ollama backend is active before the RunPod worker begins accepting jobs.
Sources: src/start.sh:3-44
Inference Engines
The worker utilizes two primary engine classes to process requests, depending on whether the input follows the standard RunPod format or the OpenAI API format.
OllamaEngine
Used for direct prompts. It wraps the input into an OpenAI-compatible structure and routes it to the internal Ollama OpenAI endpoint. If a prompt is a string, it defaults to the /v1/completions route; if it is a list, it uses /v1/chat/completions.
Sources: src/engine.py:19-50
OllamaOpenAiEngine
This engine inherits from OllamaEngine and communicates directly with the Ollama local server (typically at http://localhost:11434/v1/). It handles three main routes:
/v1/models: Lists available models./v1/chat/completions: Handles chat-based interactions./v1/completions: Handles standard completion requests.
Sources: src/engine.py:52-87
Model Pre-loading (Embedding)
To avoid downloading models at runtime, users can create a custom image with models pre-loaded. This is achieved using the MODEL_NAMES build argument.
docker build --build-arg MODEL_NAMES="llama3:8b,mistral" -t yourname/repo:tag .
Sources: embed_model/README.md:3
Preload Logic
The preload_model.sh script is used during the build process to bake models into the image layers. It starts a temporary Ollama server, iterates through the comma-separated list of models, and executes ollama pull for each.
Sources: embed_model/preload_model.sh:3-37
Request Flow
The handler.py script serves as the entry point for RunPod jobs. It determines the appropriate engine based on the presence of an openai_route in the job input.
Sources: src/handler.py:9-20, src/utils.py:1-7
Summary
The pre-built Docker image for runpod-worker-ollama provides a robust, production-ready container that bridges the RunPod serverless environment with the Ollama inference engine. By utilizing environment variables for dynamic model loading or the pre-loading scripts for static image optimization, developers can deploy a variety of LLMs with minimal configuration. The system's OpenAI-compatible interface ensures compatibility with existing tools and libraries while supporting streaming responses for real-time applications.