Introduction to Runpod Worker Ollama
Relevant source files
The following files were used as context for generating this wiki page:
Runpod Worker Ollama is a serverless runner designed to deploy and execute Ollama models on the RunPod platform. It provides a bridge between RunPod's serverless infrastructure and the Ollama API, allowing users to run large language models (LLMs) with support for standard OpenAI-compatible endpoints and streaming capabilities.
Sources: README.md:1-5, src/handler.py:30-37
The system is encapsulated within a Docker container that automates the lifecycle of the Ollama server, including model downloading, environment configuration, and request handling. It supports both direct LLM prompts and OpenAI-style chat/completion routes.
Sources: README.md:7-10, src/engine.py:23-48
System Architecture
The worker operates by wrapping the Ollama service within a RunPod serverless handler. The architecture consists of a shell-based startup sequence that initializes the Ollama backend followed by a Python-based handler that processes incoming jobs.
Component Interaction
The following diagram illustrates the interaction between the RunPod environment, the Python handler, and the internal Ollama server.
The diagram shows how a RunPod job is received by the handler, processed through the engine, and sent to the local Ollama server.
Sources: src/handler.py:10-22, src/engine.py:9-15, src/start.sh:18-25
Initialization and Startup
The worker's entry point is start.sh, which manages the initialization of the Ollama server and ensures the requested model is available before the Python handler starts receiving jobs.
- Server Initialization: The script starts the
ollama serveprocess in the background. - Health Check: It monitors
ollama.server.logfor the "Listening" signal to confirm the server is ready. - Model Management: If the
OLLAMA_MODEL_NAMEenvironment variable is set, it executesollama pullto download the model. - Worker Start: Finally, it launches the RunPod handler.
Sources: src/start.sh:15-45
Preloading Models
For optimized startup times, models can be embedded directly into the Docker image during the build process using the preload_model.sh script.
Sources: embed_model/preload_model.sh:1-35
Request Processing Logic
The handler.py script serves as the interface between RunPod and the execution engines. It utilizes the JobInput class to parse incoming JSON payloads.
Engine Selection
The worker dynamically selects an engine based on the presence of an openai_route in the input:
- OllamaOpenAiEngine: Used when
openai_routeis provided (e.g.,/v1/chat/completions). - OllamaEngine: Used for standard prompts.
Sources: src/handler.py:14-16, src/engine.py:23-40
Job Execution Flow
The sequence diagram demonstrates the flow of a single job request through the system components.
Sources: src/handler.py:10-22, src/engine.py:80-100
Configuration and Environment
The worker's behavior is governed by environment variables and specific input structures defined in the source code.
Environment Variables
| Variable Name | Description | Default Value |
|---|---|---|
OLLAMA_MODEL_NAME | The model to download and use | NULL (defaults to llama3.2:1b in engine) |
MAX_CONCURRENCY | Maximum concurrent jobs the worker handles | 8 |
Sources: README.md:14-18, src/handler.py:7-8, src/engine.py:25
Input Data Structure
The JobInput utility class extracts the following fields from the RunPod job:
llm_input: Derived frommessagesorprompt.stream: Boolean indicating if streaming is enabled.openai_route: The specific OpenAI API endpoint requested.openai_input: The raw parameters for the OpenAI-compatible request.
Sources: src/utils.py:1-7
API Support
The worker supports several OpenAI-compatible endpoints by proxying requests to the internal Ollama instance via the openai Python library.
| Endpoint | Engine Method | Description |
|---|---|---|
/v1/models | _handle_model_request | Lists available models in the local Ollama instance. |
/v1/completions | _handle_chat_or_completion_request | Standard text completion. |
/v1/chat/completions | _handle_chat_or_completion_request | Multi-turn chat completion. |
Sources: src/engine.py:59-95
Streaming Implementation
Streaming is supported for chat and completion routes. When stream is set to True, the engine yields data chunks formatted as data: {...}\n\n, concluding with data: [DONE].
Sources: src/engine.py:87-95, README.md:23-24
Summary
Runpod Worker Ollama provides a robust wrapper for running LLMs on serverless infrastructure. By leveraging a dual-engine approach, it maintains compatibility with both native Ollama prompts and the OpenAI API ecosystem. The inclusion of startup health checks and model preloading ensures high reliability and performance within the RunPod environment.
Sources: src/engine.py:18-57, src/start.sh:22-35