Streaming Response Support

Relevant source files

The following files were used as context for generating this wiki page:

• src/handler.py
• src/engine.py
• src/utils.py
• README.md
• test_inputs/openai_completion.json

Streaming Response Support in the runpod-worker-ollama project enables the worker to deliver real-time, incremental text generation updates to the client. This feature is primarily designed to support OpenAI-compatible requests for both standard completions and chat completions, providing a low-latency user experience by yielding data as it is generated by the underlying Ollama engine.

Sources: README.md:20, src/engine.py:32-35

Architecture and Data Flow

The streaming architecture utilizes Python's asynchronous generators to pipe data from the OpenAI client through the local OllamaOpenAiEngine and ultimately to the RunPod serverless handler. The workflow begins when a job is received by the handler with the stream parameter set to true.

Processing Pipeline

The system uses a layered approach to handle streaming:

1. Handler Level: The handler function uses an async for loop to iterate over batches yielded by the engine.
2. Engine Level: The OllamaEngine (or its subclass OllamaOpenAiEngine) manages the connection to the local Ollama instance (running at http://localhost:11434/v1/).
3. Response Formatting: When streaming is active, the engine formats each chunk of data into a Server-Sent Events (SSE) compatible string.

Sources: src/handler.py:11-20, src/engine.py:10-14, src/engine.py:90-99

Streaming Sequence Diagram

The following diagram illustrates the flow of a streaming request from the initial job submission to the final data chunk delivery.

The diagram shows how the OllamaOpenAiEngine acts as an intermediary, transforming Ollama's internal stream into the format expected by the RunPod handler.
Sources: src/handler.py:17-20, src/engine.py:84-100

Implementation Details

Data Structures

The JobInput class is the primary data structure used to parse and transport streaming preferences from the incoming request.

Field	Source	Description
stream	job.get("stream")	Boolean indicating if the response should be streamed. Defaults to False.
openai_route	job.get("openai_route")	The API endpoint requested (e.g., /v1/chat/completions).
llm_input	job.get("messages") or job.get("prompt")	The actual text or message list for the model.

Sources: src/utils.py:1-7

Logic Flow for Streaming

The OllamaOpenAiEngine._handle_chat_or_completion_request method contains the core logic for differentiating between synchronous and streaming responses. If stream is set to True, it enters a loop that iterates over the response object from the OpenAI client.

# src/engine.py:90-99
# If streaming is False, we can just return the response
if not openai_input.get("stream", False):
    yield response.to_dict()
    return

for chunk in response:
    # Return json of the chunk without any line breaks
    yield "data: " + json.dumps(chunk.to_dict(), separators=(',', ':')) + "\n\n"

yield "data: [DONE]"

Sources: src/engine.py:90-100

Engine Class Hierarchy

The streaming capability is distributed across specialized engine classes.

The OllamaOpenAiEngine inherits from OllamaEngine and provides the specific implementation for OpenAI-compatible streaming routes.
Sources: src/engine.py:20, src/engine.py:53

Configuration and Usage

To enable streaming, the input payload must include the stream parameter within the openai_input object or the top-level input depending on the route used.

Example Streaming Input

{
    "input": {
        "openai_route": "/v1/completions",
        "openai_input": {
            "model": "llama3.2:1b",
            "prompt": "How are you?",
            "stream": true
        }
    }
}

Sources: test_inputs/openai_completion.json:1-10, src/engine.py:35

RunPod Integration

The handler is initialized with return_aggregate_stream: True, which is a RunPod serverless configuration that allows the worker to return an asynchronous generator for streaming responses.

Sources: src/handler.py:32

Summary

Streaming Response Support in this worker provides a robust way to handle real-time LLM outputs. By leveraging the OpenAI Python client and mapping it to the local Ollama service, the worker supports standard SSE formatting (data: {...}) and correctly signals the end of a stream with the [DONE] message. This implementation ensures compatibility with front-end applications expecting OpenAI-style streaming behavior.