Serverless Handler (handler.py)

Relevant source files

The following files were used as context for generating this wiki page:

• src/handler.py
• src/engine.py
• src/utils.py
• src/start.sh
• README.md

The handler.py module serves as the primary entry point for the RunPod serverless worker. Its purpose is to bridge the RunPod infrastructure with the Ollama inference engine. It initializes the execution environment, configures concurrency settings, and defines the asynchronous request handler that processes incoming jobs by routing them to the appropriate inference engine.

This module works in tandem with the Execution Engine to handle both standard Ollama prompts and OpenAI-compatible API requests. It is typically invoked by src/start.sh after the Ollama server has been initialized and the required models have been pulled.

Sources: src/handler.py:1-32, src/start.sh:41

Core Architecture and Data Flow

The handler follows a specific lifecycle: it receives a job from RunPod, parses the input using a utility class, selects an engine based on the presence of OpenAI-specific routes, and streams the results back to the caller.

Job Lifecycle Flow

The following diagram illustrates the flow of a request from the initial RunPod trigger through the handler to the inference engine.

Component Interaction

The handler relies on the JobInput class to normalize raw job data and interacts with OllamaEngine or its subclass OllamaOpenAiEngine to perform the actual inference logic.

Component	Responsibility
runpod.serverless	Provides the worker lifecycle management and communication with the RunPod platform.
JobInput	Data structure that extracts llm_input, stream, and openai_route from the job payload.
OllamaEngine	Handles generic LLM inputs (prompts or messages) by converting them to internal formats.
OllamaOpenAiEngine	Handles OpenAI-compatible requests specifically targeting /v1/chat/completions, /v1/completions, or /v1/models.

Sources: src/handler.py:3-15, src/utils.py:1-6, src/engine.py:15-84

Configuration and Concurrency

The worker's execution behavior is influenced by environment variables and parameters passed to the RunPod start function.

Environment Variables

Variable	Description	Default Value
MAX_CONCURRENCY	Sets the maximum number of concurrent jobs the worker can handle.	8
OLLAMA_MODEL_NAME	The name of the model to download and use for inference.	NULL (llama3.2:1b in engine)

Sources: src/handler.py:7-8, README.md:15-17, src/engine.py:20

Worker Initialization

The worker is started using the runpod.serverless.start method, which configures the handler function, sets the concurrency modifier, and enables aggregate stream returns.

runpod.serverless.start(
    {
        "handler": handler,
        "concurrency_modifier": lambda x: max_concurrency,
        "return_aggregate_stream": True,
    }
)

Sources: src/handler.py:27-33

Request Processing Logic

The handler(job) function is the heart of the module. It performs the following technical steps:

1. Input Parsing: It initializes a JobInput object from job["input"].
2. Engine Selection: It checks if job_input.openai_route is set. If true, it uses OllamaOpenAiEngine; otherwise, it uses the base OllamaEngine.
3. Result Streaming: It calls engine.generate(job_input) and uses an async for loop to yield batches of data back to the RunPod stream.

Sequence Diagram: Request Handling

The sequence below shows the interaction between the handler and the underlying engines during a request.

Conclusion

The handler.py module acts as a lightweight but critical router within the runpod-worker-ollama architecture. By abstracting the engine selection and input parsing into dedicated classes, it maintains a clean interface for the RunPod serverless environment while supporting diverse API specifications, including native Ollama and OpenAI-compatible workloads.