System Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• src/handler.py
• src/engine.py
• src/utils.py
• src/start.sh
• README.md
• embed_model/preload_model.sh

This document provides a technical overview of the runpod-worker-ollama system architecture. The project is designed to wrap the Ollama inference engine within a RunPod serverless worker environment, allowing for scalable, on-demand Large Language Model (LLM) execution with OpenAI-compatible API support.

The system facilitates model preloading, environment configuration, and request handling through a tiered architecture that bridges RunPod's serverless infrastructure with local Ollama service instances.

Core Components

The architecture consists of three primary layers: the environment initialization layer, the request handling layer, and the execution engines.

1. Initialization and Environment

The system begins with a shell-based startup sequence that ensures the local Ollama server is running and the required models are available before the Python worker starts.

2. Request Handling (Handler Layer)

The handler.py script serves as the entry point for RunPod jobs. It utilizes the runpod library to manage concurrency and stream responses back to the client.

Component	Responsibility	Source
handler	Coroutine that parses JobInput and selects the appropriate engine.	src/handler.py:9-20
JobInput	Data class mapping raw job dictionaries to structured attributes like llm_input and openai_route.	src/utils.py:1-6
concurrency_modifier	Limits active jobs based on MAX_CONCURRENCY environment variable (default: 8).	src/handler.py:6-7, 34

3. Execution Engines

The system employs two engine classes to handle different request formats. OllamaOpenAiEngine inherits from OllamaEngine to provide specialized handling for OpenAI-compatible endpoints.

Data Flow and API Logic

The system translates incoming JSON payloads into internal JobInput objects, which are then routed based on the presence of an openai_route.

Sequence of a Request

Engine Logic and Routing

• Standard Input: If openai_route is missing, the OllamaEngine converts the llm_input (string or list) into an OpenAI-compatible completion or chat completion request.
• OpenAI Input: If openai_route is provided (e.g., /v1/models, /v1/completions), the OllamaOpenAiEngine bypasses internal transformations and communicates directly with the local Ollama instance using the openai Python client.
• Streaming: The engine supports real-time streaming by yielding chunks in the data: {...} \n\n format, ending with data: [DONE].

Sources: src/engine.py:56-101, src/utils.py:1-6

Configuration and Deployment

Environment Variables

Variable	Type	Default	Description
OLLAMA_MODEL_NAME	String	llama3.2:1b	The model to be pulled and used for inference.
MAX_CONCURRENCY	Integer	8	Maximum number of concurrent jobs the worker will process.

Sources: README.md:15-19, src/handler.py:6-7, src/engine.py:22-23

Preloading Models

To reduce startup latency, models can be baked into the Docker image using the preload_model.sh script. This script starts a temporary Ollama server, pulls the specified MODEL_NAMES, and then shuts down the server.

# Example from preload_model.sh
ollama serve &
OLLAMA_PID=$!
sleep 5
IFS=',' read -r -a MODELS <<< "$MODEL_NAMES"
for MODEL_NAME in "${MODELS[@]}"; do
  ollama pull "$MODEL_NAME"
done
kill $OLLAMA_PID

Sources: embed_model/preload_model.sh:4-36, embed_model/README.md:3-3

Summary

The runpod-worker-ollama architecture provides a robust bridge between RunPod serverless and Ollama. By utilizing a local sidecar-style Ollama process and a Python-based request translator, it enables high-performance LLM inference with support for both native Ollama interactions and standardized OpenAI API calls, complete with streaming capabilities and configurable concurrency.