Getting Started Guide
Relevant source files
The following files were used as context for generating this wiki page:
The RunPod Worker Ollama project provides a serverless environment for running Ollama models on the RunPod platform. It acts as a bridge between RunPod's serverless infrastructure and the Ollama API, supporting both standard LLM prompts and OpenAI-compatible API routes for completions and chat.
This guide covers the initial setup, configuration via environment variables, request formats, and the internal architecture of the worker.
Sources: README.md:1-5, src/engine.py:10-25
Deployment and Setup
To get started, deploy the worker using the Docker image svenbrnn/runpod-ollama:latest on the RunPod serverless console. The worker is designed to automatically handle model management and volume mounting.
Sources: README.md:5-9
Environment Variables
Configuration is primarily handled through environment variables defined at the pod level.
| Variable Name | Description | Default Value |
|---|---|---|
OLLAMA_MODEL_NAME | The name of the model to download and use (e.g., llama3.2:1b). | NULL (Defaults to llama3.2:1b in code) |
MAX_CONCURRENCY | Sets the maximum number of concurrent jobs the worker handles. | 8 |
Sources: README.md:14-18, src/engine.py:27, src/handler.py:6-7
Initial Startup Sequence
When the container starts, the start.sh script initializes the environment by starting the Ollama server and ensuring the requested model is available.
The diagram shows the initialization flow from process cleanup to model verification before starting the Python worker.
Sources: src/start.sh:3-44
Execution Architecture
The worker utilizes two primary engine classes to process requests: OllamaEngine for basic prompts and OllamaOpenAiEngine for requests targeting OpenAI-compatible endpoints.
Request Handling Flow
The handler.py script serves as the entry point, receiving jobs from RunPod and routing them to the appropriate engine.
This sequence illustrates how a request is parsed and proxied to the local Ollama instance.
Sources: src/handler.py:9-25, src/engine.py:10-15, src/utils.py:1-6
Input Data Structures
The worker uses the JobInput class to normalize incoming JSON payloads.
- Standard Prompt: Uses
messagesorpromptfields. - OpenAI Route: Triggered if
openai_routeis present (e.g.,/v1/chat/completions). - Streaming: Controlled by the
streamboolean flag.
Sources: src/utils.py:1-6, src/engine.py:30-45
Advanced Usage
Model Preloading
To reduce cold start times, models can be baked into the Docker image during the build process using the preload_model.sh script.
Build Command:
docker build --build-arg MODEL_NAMES="llama3.2:1b,mistral" -t <yourname>/<yourrepo>:<tag> .
Sources: embed_model/README.md:3-3, embed_model/preload_model.sh:1-35
Supported OpenAI Routes
The OllamaOpenAiEngine handles the following specific endpoints:
/v1/models: Lists available models viaclient.models.list()./v1/chat/completions: Processes chat messages./v1/completions: Processes raw text prompts.
Sources: src/engine.py:68-80
Summary
The RunPod Worker Ollama simplifies the deployment of Ollama models in a serverless environment by automating process management and providing an OpenAI-compatible interface. By configuring OLLAMA_MODEL_NAME and optionally preloading models, developers can achieve rapid deployment of various LLMs with support for both synchronous and streaming responses.