Container Startup Process (start.sh)
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The container startup process is managed primarily by src/start.sh, which serves as the entry point for the RunPod serverless worker. Its purpose is to orchestrate the initialization of the Ollama server, manage model dependencies, and finally launch the RunPod handler that processes incoming inference requests.
This process ensures that the local Ollama environment is fully operational and the required Large Language Model (LLM) is available before the worker starts accepting jobs. It handles environment cleanup, process monitoring, and provides a bridge between the RunPod infrastructure and the underlying Ollama engine.
Sources: src/start.sh, README.md
Initialization and Cleanup
Upon execution, the startup script establishes a controlled environment by cleaning up existing processes and setting up signal traps. This prevents conflicts from previous container runs or unexpected terminations.
Process Management
The script immediately attempts to kill any existing ollama processes using pgrep and xargs. Furthermore, it defines a cleanup() function that uses pkill -P $$ to terminate all child processes of the current script when a SIGINT or SIGTERM signal is received.
Sources: src/start.sh:3-15
Ollama Server Orchestration
The core of the startup process is the management of the Ollama server, which acts as the backend for LLM operations.
Server Lifecycle
The script starts the Ollama server in the background, redirecting both standard output and error to a log file (ollama.server.log) for monitoring. The process ID (PID) is captured to allow for tracking. The script then enters a loop, polling the log file for the string "Listening" to confirm the server is ready to accept connections.
Sources: src/start.sh:17-33
The following diagram illustrates the sequence of operations during server startup:
The startup flow ensures the server is "Listening" before proceeding to model management or worker execution.
Sources: src/start.sh:1-42
Model Provisioning
The system supports two methods of ensuring a model is available: dynamic pulling at startup or pre-embedding during the Docker build process.
Dynamic Pulling
If the environment variable OLLAMA_MODEL_NAME is provided, the startup script executes ollama pull for that specific model. If the variable is empty, this step is skipped, assuming the model might already be present or will be handled differently.
Sources: src/start.sh:35-40, README.md:9-13
Build-time Preloading
For faster cold starts, the project provides embed_model/preload_model.sh. This script is intended to be used during the Docker build process to bake models into the image. It handles multiple models by splitting a comma-separated list provided in the MODEL_NAMES argument.
Sources: embed_model/preload_model.sh:1-32, embed_model/README.md:3-3
| Configuration Variable | Description | Source |
|---|---|---|
OLLAMA_MODEL_NAME | The name of the model to download at runtime. | README.md:13 |
MODEL_NAMES | Comma-separated list of models to preload during build. | embed_model/preload_model.sh:17 |
Worker Handover
Once the Ollama server is verified and models are prepared, the script executes the RunPod handler.
python -u handler.py $1
Sources: src/start.sh:42
The handler.py script initializes the RunPod serverless environment, setting the maximum concurrency (defaulting to 8) and mapping incoming requests to the OllamaEngine or OllamaOpenAiEngine.
Sources: src/handler.py:6-30
Process Relationships
The startup process establishes a hierarchy where the shell script manages the background Ollama process and the foreground Python worker.
The Python handler communicates with the Ollama server via the local engine (OpenAI compatible) at http://localhost:11434/v1/.
Sources: src/engine.py:8-12, src/start.sh:18
Summary
The start.sh process is a critical orchestration layer that transforms a generic container into a specialized RunPod worker. By handling process cleanup, server verification, and model pulling, it ensures a robust environment for handler.py to interact with the OllamaEngine. This architecture allows for flexible deployments where models can be either baked into the image for speed or specified via environment variables for versatility.