Testing OpenAI Models Discovery

Relevant source files

The following files were used as context for generating this wiki page:

• test_inputs/openai_get_models.json
• src/engine.py
• src/handler.py
• src/utils.py
• src/start.sh
• README.md

Introduction

Testing OpenAI Models Discovery refers to the process of verifying the system's ability to list available Large Language Models (LLMs) via an OpenAI-compatible API endpoint. This functionality is crucial for client applications that need to dynamically discover which models are currently served by the underlying Ollama instance within the RunPod serverless environment.

The system acts as a proxy, translating incoming OpenAI-style requests into calls for the local Ollama service. The discovery mechanism specifically targets the /v1/models route, ensuring that the worker can report its capabilities accurately to the RunPod orchestrator or direct API consumers.

Sources: src/engine.py:80-84, test_inputs/openai_get_models.json

Request Architecture and Flow

The discovery process begins with a JSON payload submitted to the RunPod worker. This payload specifies an openai_route pointing to the models endpoint. The worker's handler identifies the appropriate engine to process the request based on the presence of this route.

Model Discovery Sequence

The following diagram illustrates the flow from the initial test input to the retrieval of model data from the Ollama backend:

Components and Logic

1. Job Parsing: The JobInput class extracts the openai_route and openai_input from the raw RunPod job.
   • Sources: src/utils.py:1-7
2. Engine Routing: In src/handler.py, the system checks if job_input.openai_route is present. If so, it instantiates OllamaOpenAiEngine.
   • Sources: src/handler.py:15-16
3. Route Handling: Inside OllamaOpenAiEngine.generate, the system specifically checks for the /v1/models string to trigger the model listing logic.
   • Sources: src/engine.py:80-83

Data Structures and API Definition

The model discovery test utilizes a specific JSON structure to simulate a RunPod environment trigger.

Test Input Configuration

The test input file openai_get_models.json provides the necessary structure to exercise the discovery logic.

Field	Type	Description
input	Object	Root container for the RunPod job data.
openai_route	String	Must be set to /v1/models for discovery.
openai_input	Object	Container for OpenAI parameters (empty for model listing).

Sources: test_inputs/openai_get_models.json:1-7

Implementation Detail: _handle_model_request

The core logic for fetching models is implemented in the _handle_model_request method of OllamaOpenAiEngine. It utilizes the openai Python library configured to point to the local Ollama instance.

async def _handle_model_request(self):
    try:
        response = client.models.list()
        # build a json response from the response object
        yield {"object": "list", "data": [model.to_dict() for model in response.data]} 
    except Exception as e:
        yield {"error": str(e)}

Sources: src/engine.py:88-94

Environment and Runtime Requirements

For model discovery to succeed during testing, the worker must have a functioning Ollama server reachable at http://localhost:11434/v1/.

• Initialization: The src/start.sh script is responsible for starting the Ollama server and waiting for the "Listening" status before the handler begins processing jobs.
• Model Availability: The list returned depends on models pulled via ollama pull or specified in the OLLAMA_MODEL_NAME environment variable.

Sources: src/start.sh:17-36, src/engine.py:12, README.md:9-13

Conclusion

Testing OpenAI Models Discovery ensures that the RunPod worker correctly identifies and translates OpenAI-compatible requests into actionable commands for the local Ollama service. By using the OllamaOpenAiEngine, the system provides a seamless interface for model metadata retrieval, allowing external tools to interact with the serverless worker as if it were a native OpenAI endpoint.

Sources: src/engine.py:76-85, src/handler.py:15-18