Local Development Setup

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/engine.py
• src/handler.py
• src/start.sh
• src/utils.py
• embed_model/preload_model.sh
• src/requirements.txt

Introduction

Local development for the RunPod Ollama worker involves configuring a Python environment capable of interacting with a local Ollama server and simulating the RunPod serverless lifecycle. The environment integrates the Ollama binary for model serving with a Python-based handler that wraps the inference logic for RunPod's infrastructure.

The setup primarily facilitates the execution of LLM tasks via two pathways: direct Ollama interaction and an OpenAI-compatible API layer. Developers can test these paths locally using provided JSON input structures to ensure the worker correctly processes prompts, manages model preloading, and handles streaming responses.

Sources: README.md:1-10, src/handler.py:1-25

Environment Configuration

The project relies on specific environment variables and Python dependencies to function. Local setup requires a .env file to be loaded by the engine to manage model selection and concurrency.

Key Environment Variables

Variable Name	Description	Default Value
OLLAMA_MODEL_NAME	The name of the model to download/use	NULL (Defaults to llama3.2:1b in code)
MAX_CONCURRENCY	Maximum concurrent jobs for the RunPod handler	8

Sources: README.md:15-19, src/engine.py:27, src/handler.py:6-7

Dependency Management

The worker requires the following Python packages for local development, which can be installed via pip install -r src/requirements.txt:

• runpod: For serverless integration.
• python-dotenv: For managing environment variables.
• openai: For the OpenAI-compatible API interface.
• orjson: For high-performance JSON parsing.

Sources: src/requirements.txt:1-4

System Architecture and Workflow

The system architecture consists of a shell-based initialization process that coordinates the Ollama background service and the Python worker.

Initialization Flow

The start.sh script manages the lifecycle of the local environment, ensuring the Ollama server is active before starting the Python handler.

The diagram above illustrates the sequence of operations performed by the entrypoint script to prepare the local environment.
Sources: src/start.sh:3-44

Data Flow Logic

When a request is received, the JobInput class parses the data, and the OllamaEngine determines whether to route it through standard completion or the OpenAI-compatible interface.

This sequence shows how handler.py acts as the orchestrator between the input and the underlying OllamaEngine classes.
Sources: src/handler.py:9-20, src/engine.py:24-60

Model Preloading

For optimized local testing and container builds, the project includes a mechanism to bake models into the environment. This is handled by embed_model/preload_model.sh.

Preloading Steps

1. Initialize Server: The script starts ollama serve in the background.
2. Wait for Availability: A sleep period ensures the server is ready.
3. Iterative Pulling: It splits the MODEL_NAMES comma-separated string and executes ollama pull for each.
4. Cleanup: The Ollama process is killed once pulling is successful.

Sources: embed_model/preload_model.sh:3-33

Testing the Local Setup

Local testing is conducted using JSON files that mimic RunPod's input structure. The worker expects an input object containing either a raw prompt or openai_input.

Input Structure Example

The JobInput class in src/utils.py extracts fields to determine execution logic.

# src/utils.py:1-6
class JobInput:
    def __init__(self, job):
        self.llm_input = job.get("messages", job.get("prompt"))
        self.stream = job.get("stream", False)
        self.openai_route = job.get("openai_route")
        self.openai_input = job.get("openai_input")

Sources: src/utils.py:1-6, src/test_input.json:1-5

Testing Routes

Test File	Description	Route Targeted
openai_completion.json	Tests standard completion via OpenAI API	/v1/completions
openai_get_models.json	Retrieves list of available models	/v1/models
test_input.json	Tests basic prompt functionality	Default (Internal)

Sources: test_inputs/openai_completion.json:1-10, test_inputs/openai_get_models.json:1-6, src/test_input.json:1-5

Conclusion

The local development setup for the RunPod Ollama worker provides a bridge between RunPod's serverless requirements and the Ollama inference engine. By utilizing start.sh for process management and the OllamaOpenAiEngine for API compatibility, developers can verify streaming and completion logic locally before deploying to production infrastructure. The inclusion of preloading scripts further allows for the creation of self-contained environments with embedded models.