Request Lifecycle and Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• src/handler.py
• src/engine.py
• src/utils.py
• src/start.sh
• README.md
• embed_model/preload_model.sh

The Request Lifecycle and Data Flow in this project describes the journey of an inference request from its arrival at the RunPod serverless worker to its execution via an internal Ollama instance. The system acts as a bridge, accepting standard RunPod job inputs or OpenAI-compatible payloads and translating them into requests for the Ollama local API.

The architecture is designed to handle both standard completion and chat completion requests, supporting both synchronous and streaming responses. This is achieved through a specialized engine hierarchy that abstracts the communication with the underlying Ollama service, which runs locally within the container environment.

Sources: src/handler.py:10-23, src/engine.py:16-25, README.md:1-15

System Initialization

Before any requests are processed, the environment must be initialized. This involves starting the Ollama server and ensuring the required Large Language Model (LLM) is available.

The initialization process follows these steps:

1. Server Start: The start.sh script launches the ollama serve process in the background.
2. Health Check: It monitors ollama.server.log until the "Listening" string appears, indicating readiness.
3. Model Loading: If the OLLAMA_MODEL_NAME environment variable is set, the worker performs an ollama pull to fetch the model from the Ollama library.
4. Worker Registration: Finally, the Python handler is started, which registers the worker with the RunPod serverless infrastructure.

Sources: src/start.sh:11-41, README.md:5-15

The diagram above illustrates the sequence of operations required to prepare the worker for processing requests. Sources: src/start.sh:11-41

Request Ingress and Parsing

Requests enter the system through the handler function, which is the entry point for the RunPod serverless worker. The worker uses a JobInput utility class to normalize the incoming JSON payload.

Input Normalization

The JobInput class extracts key fields from the job input, regardless of whether the request uses a custom format or an OpenAI-compatible structure.

Field	Description	Source
llm_input	The actual prompt or message list. Derived from messages or prompt.	src/utils.py:3
stream	Boolean indicating if the response should be streamed. Defaults to False.	src/utils.py:4
openai_route	The specific API endpoint targeted (e.g., /v1/chat/completions).	src/utils.py:5
openai_input	The raw configuration for OpenAI-style requests.	src/utils.py:6

Sources: src/utils.py:1-6, src/handler.py:10-15

Engine Selection and Execution

The system uses a strategy pattern to determine how to process a request based on the presence of the openai_route field.

Selection Logic

• OllamaOpenAiEngine: Selected if openai_route is provided. This engine handles structured OpenAI-compatible requests, including model listing, chat, and completions.
• OllamaEngine: Selected for standard RunPod inputs. It wraps the raw input into an internal OpenAI-style job before passing it to the processing logic.

Sources: src/handler.py:16-17, src/engine.py:27-46

Execution Flow

1. Instatiation: The selected engine class is instantiated.
2. Generation: The generate() method is called, which returns an asynchronous generator.
3. Client Interaction: The engine uses an OpenAI client configured with base_url='http://localhost:11434/v1/' to communicate with the local Ollama server.

Sources: src/handler.py:16-23, src/engine.py:8-13, src/engine.py:49-65

This sequence diagram shows the internal communication between the RunPod handler, the internal engines, and the local Ollama server. Sources: src/handler.py:10-23, src/engine.py:49-65

Data Processing and Response Handling

The OllamaOpenAiEngine handles the logic for various OpenAI routes and manages the format of the output.

Supported Routes

Route	Logic
/v1/models	Calls client.models.list() and returns a list of available models.
/v1/chat/completions	Calls client.chat.completions.create. Supports streaming.
/v1/completions	Calls client.completions.create. Supports streaming.

Sources: src/engine.py:56-65

Streaming vs. Non-Streaming

• Non-Streaming: The engine waits for the full response from Ollama, converts the model object to a dictionary using .to_dict(), and yields it.
• Streaming: The engine iterates over chunks from the OpenAI client. For each chunk, it yields a string formatted as data: <JSON>\n\n. When the stream ends, it yields data: [DONE].

Sources: src/engine.py:77-94

Summary of Data Flow

The request lifecycle is a linear transition from external payload to local API call. Data is first normalized into a JobInput object, routed to the appropriate OllamaEngine subclass, and then converted into a call against the local Ollama REST API. The results are streamed back to the caller in real-time if requested, or returned as a single aggregate response.

Sources: src/handler.py:10-23, src/engine.py:49-94