Testing OpenAI Completion Emulation

Testing OpenAI Completion Emulation involves verifying the worker's ability to intercept and process requests formatted for the OpenAI API and translate them into commands compatible with an underlying Ollama instance. This feature allows users to leverage existing OpenAI-compatible client libraries while executing models locally or in a serverless RunPod environment.

The system uses a wrapper architecture where requests containing specific "openai_route" keys are routed through a specialized engine that interacts with a local Ollama server via the OpenAI Python library. This emulation supports standard completion, chat completion, and model listing endpoints.
Sources: src/engine.py:11-15, README.md:19-21

Architecture and Data Flow

The emulation process begins at the handler.py entry point, which determines whether to use the standard OllamaEngine or the OllamaOpenAiEngine based on the presence of an openai_route in the job input.

The flow ensures that any request targeting the OpenAI-compatible paths is handled by the specialized emulation layer.
Sources: src/handler.py:12-14, src/engine.py:11-15

Request Translation Logic

When a request is identified as an OpenAI-style request, it is encapsulated in a JobInput object. The OllamaOpenAiEngine then inspects the openai_route to decide how to interface with the local Ollama backend.

Sources: src/engine.py:65-73, src/utils.py:1-7

Implementation Details

The emulation is implemented primarily within src/engine.py. It utilizes the official openai Python package, configured to point at the local Ollama service.

The OllamaOpenAiEngine Class

This class inherits from OllamaEngine and overrides the generate method to handle structured OpenAI payloads. It manages both synchronous and streaming responses.

Streaming is explicitly supported for OpenAI requests, providing real-time token delivery to the user.
Sources: src/engine.py:90-108, README.md:21-21

Data Structures for Emulation

The JobInput utility class standardizes the input format. When emulation is active, it specifically extracts openai_route and openai_input.

class JobInput:
    def __init__(self, job):
        self.llm_input = job.get("messages", job.get("prompt"))
        self.stream = job.get("stream", False)
        self.openai_route = job.get("openai_route")
        self.openai_input = job.get("openai_input")

Sources: src/utils.py:1-7

Test Input Configurations

To verify the emulation, the project includes specific JSON templates used for testing the completion and model listing endpoints.

Completion Emulation Test

The following configuration tests the /v1/completions endpoint, providing a model name and a prompt within the expected OpenAI schema.

{
    "input": {
        "openai_route": "/v1/completions",
        "openai_input": {
            "model": "llama3.2:1b",
            "prompt": "How are you?",
            "max_tokens": 60
        }
    }
}

Sources: test_inputs/openai_completion.json:1-10

Model List Emulation Test

Testing the model retrieval ensures the emulation correctly maps Ollama's local model library to the OpenAI model list format.

{
    "input": {
        "openai_route": "/v1/models",
        "openai_input": {}
    }
}

Sources: test_inputs/openai_get_models.json:1-6

Summary

Testing the OpenAI Completion Emulation ensures that the worker acts as a transparent proxy for Ollama. By utilizing the OllamaOpenAiEngine and standardized test inputs, the system confirms that OpenAI-formatted requests for completions, chat, and model metadata are correctly translated and executed by the local Ollama service.
Sources: src/engine.py:60-110, src/handler.py:10-20