Preload Script Breakdown (preload_model.sh)

Introduction

The preload_model.sh script is a utility designed to bake Large Language Models (LLMs) directly into the Docker image during the build process. By preloading models, the worker avoids the latency of downloading heavy model weights at runtime, which is critical for serverless environments where cold-start times must be minimized. This script complements the standard runtime behavior where models are otherwise pulled dynamically based on environment variables.

Sources: embed_model/preload_model.sh, embed_model/README.md, README.md:12-14

Preload Workflow and Logic

The script orchestrates a local Ollama server instance specifically to handle model retrieval. It starts the server, waits for it to become responsive, iterates through a list of provided model names, and ensures each is successfully downloaded to the local filesystem before terminating the server.

Process Flow

The following diagram illustrates the sequence of operations within the preloading script:

Build-Time Integration

To utilize this script, the project requires the MODEL_NAMES build argument during the Docker build phase. This allows developers to specify one or more models (comma-separated) to be embedded in the final image.

Sources: embed_model/preload_model.sh:7-14, embed_model/README.md:3

Comparison with Runtime Initialization

While preload_model.sh is used during image construction, the project also contains src/start.sh, which handles model verification at runtime. If a model was not preloaded, start.sh uses the OLLAMA_MODEL_NAME environment variable to pull the required model when the container starts.

Key Implementation Details

Server Lifecycle Management

The script uses standard Linux signals to manage the ollama serve process. The PID is captured using $! immediately after starting the server in the background. If any model pull fails, the script ensures the server is terminated before exiting with an error code to fail the Docker build.

# Capture the PID of the Ollama server
OLLAMA_PID=$!

# ... logic ...

if ollama pull "$MODEL_NAME"; then
    echo "Successfully pulled model: $MODEL_NAME"
  else
    echo "Failed to pull model: $MODEL_NAME"
    kill $OLLAMA_PID
    exit 1
fi

Sources: embed_model/preload_model.sh:7-26

Model Parsing

The script handles multiple models by splitting the MODEL_NAMES string using the Internal Field Separator (IFS) and reading it into a Bash array. This enables the preloading of multiple specialized models into a single worker image.

# Split the comma-separated model names into an array
IFS=',' read -r -a MODELS <<< "$MODEL_NAMES"

Sources: embed_model/preload_model.sh:17-18

Conclusion

The preload_model.sh script is a critical build-time component that optimizes the RunPod worker by embedding necessary Ollama models. By managing the server lifecycle and handling errors during the pull process, it ensures that the resulting Docker container is ready for immediate inference upon deployment.

Sources: embed_model/preload_model.sh, README.md:12-14