Embedding Models into the Docker Image
Relevant source files
The following files were used as context for generating this wiki page:
Embedding models into the Docker image is a strategy used in the runpod-worker-ollama project to pre-package Large Language Models (LLMs) directly within the container artifact. This approach eliminates the need for the worker to download heavy model files during the initial cold start on RunPod serverless infrastructure, significantly reducing the time to readiness for inference tasks.
The process involves using Docker build arguments to specify which models should be pulled from the Ollama library during the image creation phase. By preloading these models, the container's file system already contains the necessary weights and configurations, allowing the Ollama server to serve requests immediately upon startup.
Sources: embed_model/README.md:1-5, README.md:21-23
Preloading Architecture and Workflow
The preloading mechanism relies on a build-time script that orchestrates the initialization of a local Ollama server, fetches the requested models, and then shuts down cleanly to persist the changes in the Docker layer.
The Preloading Script (preload_model.sh)
The script preload_model.sh is the core component responsible for the embedding process. It performs the following sequence:
- Starts the Ollama server in the background using
ollama serve. - Waits for the server to become responsive.
- Parses the
MODEL_NAMESenvironment variable, which contains a comma-separated list of models. - Executes
ollama pullfor each model specified. - Terminates the background Ollama process once all pulls are complete.
Sources: embed_model/preload_model.sh:3-37
Build Process Flow
The following diagram illustrates the flow of embedding a model during the Docker image build process:
The build process ensures that if any model fail to download, the entire build fails, preventing the creation of an incomplete image.
Sources: embed_model/preload_model.sh:3-37, embed_model/README.md:3-5
Build Configuration
To embed models, the user must provide the MODEL_NAMES build argument during the docker build command.
| Argument | Description | Example Value |
|---|---|---|
MODEL_NAMES | Comma-separated list of Ollama models to download during build | llama3.2:1b,phi3 |
-t | Tag for the resulting Docker image | myrepo/runpod-ollama:custom |
Sources: embed_model/README.md:3-5, embed_model/preload_model.sh:16-18
Build Command Example
docker build --build-arg MODEL_NAMES="llama3.2:1b" -t <yourname>/<yourrepo>:<tag> .
Sources: embed_model/README.md:3-5
Runtime Interaction with Embedded Models
Once an image with embedded models is deployed, the system behaves differently depending on the environment variables provided at runtime.
Model Selection Logic
The OllamaEngine in src/engine.py retrieves the model name from the OLLAMA_MODEL_NAME environment variable. If this variable is not set, it defaults to llama3.2:1b. This model name is then passed to the OpenAI-compatible API routes.
Sources: src/engine.py:24-26, README.md:7-11
Startup Synchronization
The src/start.sh script manages the runtime execution. It checks if OLLAMA_MODEL_NAME is provided. If the model was already embedded via the build process, the ollama pull command in start.sh will detect the existing files and complete instantly.
Sources: src/start.sh:17-43, src/handler.py:34-40
Comparison: Embedded vs. Dynamic Pull
| Feature | Embedded Model (Build-time) | Dynamic Pull (Runtime) |
|---|---|---|
| Start Speed | Fast (Models local to image) | Slow (Must download on first start) |
| Image Size | Large (Includes weights) | Small (Weights excluded) |
| Configuration | Uses --build-arg MODEL_NAMES | Uses OLLAMA_MODEL_NAME env |
| Reliability | High (Network issues handled at build) | Medium (Dependent on runtime network) |
Sources: README.md:7-13, embed_model/README.md:3-5, src/start.sh:37-41
Conclusion
Embedding models into the Docker image via the embed_model utility is the recommended approach for production deployments where low latency and cold-start minimization are critical. By utilizing the preload_model.sh script during the Docker build phase, developers can ensure that the runpod-worker-ollama container is fully self-contained and ready for immediate inference upon deployment.