Ollama Engine Integration (engine.py)

Relevant source files

The following files were used as context for generating this wiki page:

• src/engine.py
• src/handler.py
• src/utils.py
• src/start.sh
• README.md

The Ollama Engine Integration serves as the core processing logic for the RunPod worker, bridging the gap between RunPod job requests and the local Ollama service. It translates incoming requests into OpenAI-compatible formats, allowing the worker to leverage the OpenAI Python client to communicate with Ollama's local API endpoint.

This system is designed to handle both standard LLM prompts and structured OpenAI-style requests, supporting features like streaming responses and model management. It ensures that the worker can dynamically select models based on environment configurations while maintaining a consistent interface for the RunPod serverless environment.
Sources: src/engine.py:1-25, README.md:1-15

Core Engine Architecture

The architecture consists of two primary classes: OllamaEngine and OllamaOpenAiEngine. The OllamaEngine acts as a high-level router that prepares incoming JobInput objects for processing, while OllamaOpenAiEngine (which inherits from OllamaEngine) handles the specific logic for communicating with the OpenAI-compatible API provided by Ollama.

Class Hierarchy and Relationships

The following class diagram illustrates the relationship between the engine classes and the utility data structures.

The system uses JobInput to encapsulate request data, supporting both direct prompts (mapped to /v1/completions) and chat messages (mapped to /v1/chat/completions).
Sources: src/engine.py:14-68, src/utils.py:1-6

Request Flow and Data Processing

When a job is received by the worker, it is processed through a defined sequence of steps starting from the shell script initialization to the Python handler and finally the engine.

Data Flow Diagram

This flowchart describes the lifecycle of a request from the initial input to the streaming output.

The engine initialization involves loading environment variables via python-dotenv. If a request does not specify an openai_route, the OllamaEngine automatically wraps the input into an OpenAI-compatible structure, defaulting the model to llama3.2:1b if Ollama_MODEL_NAME is not set.
Sources: src/engine.py:22-45, src/handler.py:10-18

Execution Sequence

The sequence below shows how the OllamaOpenAiEngine interacts with the Ollama local server using the OpenAI client.

API Implementation Details

The engine handles three primary routes compatible with the OpenAI specification. These are processed within OllamaOpenAiEngine._handle_chat_or_completion_request and OllamaOpenAiEngine._handle_model_request.

Supported Routes and Operations

Route	Method	Description	Source
/v1/models	GET	Returns a list of available models pulled into the Ollama instance.	src/engine.py:70-76
/v1/chat/completions	POST	Handles structured message-based chat interactions.	src/engine.py:80-100
/v1/completions	POST	Handles legacy single-prompt completions.	src/engine.py:80-100

Input Configuration

The JobInput class parses the incoming RunPod JSON payload.

Field	Type	Description
llm_input	String/List	The prompt or message list.
stream	Boolean	Whether to stream results (default: False).
openai_route	String	The target OpenAI endpoint.
openai_input	Dictionary	The raw parameters for the OpenAI client.
Sources: src/utils.py:1-6, src/engine.py:56-65

Environment and Infrastructure

The engine relies on the Ollama server running locally within the container. The start.sh script ensures the server is ready before the Python engine attempts to connect.

• Base URL: http://localhost:11434/v1/
• Default Model: llama3.2:1b (configurable via OLLAMA_MODEL_NAME)
• Concurrency: Managed via MAX_CONCURRENCY environment variable (default: 8).

The server startup sequence is critical for the engine's functionality:

1. ollama serve is started in the background.
2. The script waits for the "Listening" log entry in ollama.server.log.
3. If OLLAMA_MODEL_NAME is provided, the model is pulled before the worker starts.
   Sources: src/start.sh:15-38, src/engine.py:8-12, src/handler.py:6-7

Summary

The Ollama Engine Integration provides a robust abstraction layer that allows RunPod workers to function as OpenAI-compatible endpoints while running local Ollama instances. By utilizing an inheritance-based engine structure, the system seamlessly handles both generic prompt inputs and specialized OpenAI API calls, supporting real-time streaming and dynamic model management.