Troubleshooting and Error Handling
Relevant source files
The following files were used as context for generating this wiki page:
Troubleshooting and error handling in the Runpod Ollama worker are managed through a combination of shell-level process monitoring, try-except blocks in the Python runtime, and explicit validation of API routes. The system is designed to handle failures during initialization (such as model pulling) and runtime execution (such as invalid OpenAI API routes or connection issues with the local Ollama server).
The architecture relies on the OllamaOpenAiEngine to catch exceptions during inference and model listing, returning structured error messages to the Runpod handler. This ensures that even when a request fails, the worker provides a JSON-formatted error response instead of crashing the entire container.
Startup and Lifecycle Troubleshooting
Errors during the container startup are primarily handled within the initialization scripts. These scripts manage the Ollama server process and ensure required models are available before the Python handler starts processing jobs.
Server Initialization Checks
The start.sh script monitors the Ollama server startup by checking the ollama.server.log for a "Listening" status. If the server fails to start, the script enters a wait loop.
The script also implements a cleanup function that traps SIGINT and SIGTERM signals to ensure child processes are terminated gracefully.
Sources: src/start.sh:3-44
Model Preloading Failures
When building an image with preloaded models, the preload_model.sh script includes error handling for failed model downloads. If the ollama pull command fails for any specific model, the script terminates the Ollama server and exits with an error code to fail the build process.
Sources: embed_model/preload_model.sh:22-26
Runtime Error Handling
Runtime errors are caught within the OllamaOpenAiEngine class using try...except blocks. These blocks wrap the interactions with the OpenAI client which communicates with the local Ollama instance.
Error Response Format
When an exception occurs during a request, the engine yields a dictionary containing an error key with the exception message. This allows the Runpod handler to return a clear message to the user.
| Error Type | Trigger Condition | Response Format |
|---|---|---|
| Invalid Route | Request sent to an unsupported openai_route | {"error": "Invalid route"} |
| Model List Failure | Local Ollama server connection error or listing error | {"error": "<exception_string>"} |
| Inference Failure | Errors during chat or completion generation | {"error": "<exception_string>"} |
Sources: src/engine.py:72-110
Request Flow and Validation
The system validates the input structure through the JobInput class and selects the appropriate engine. If the openai_route provided in the job input does not match expected endpoints, the OllamaOpenAiEngine returns an explicit error.
Sources: src/engine.py:65-72, src/handler.py:10-18
Component-Specific Troubleshooting
OpenAI Client Configuration
The OpenAI client is hardcoded to connect to http://localhost:11434/v1/. Issues here typically arise if the Ollama server (started in start.sh) crashes or fails to bind to the expected port.
Sources: src/engine.py:9-14
Input Data Validation
The JobInput class in utils.py handles extraction of inputs. Troubleshooting data issues involves verifying that the input field in the Runpod request contains either prompt, messages, or the openai_input object.
Sources: src/utils.py:1-6
Concurrency Limits
The worker uses an environment variable MAX_CONCURRENCY (defaulting to 8) to control the number of simultaneous jobs. If the worker becomes unresponsive under load, this value may need adjustment.
Sources: src/handler.py:6-7
Summary
Troubleshooting the Runpod Ollama worker involves verifying the Ollama server logs during startup and checking the JSON error responses provided by the OllamaOpenAiEngine. Error handling is implemented defensively at the engine level to capture OpenAI client exceptions and route mismatches, ensuring the worker remains stable even when receiving malformed requests or encountering internal connectivity issues between the Python wrapper and the Ollama backend.