Docker Deployment and Hardening
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
LiteLLM provides a robust framework for containerized deployment, focusing on ease of use for development and rigorous security for production. The system utilizes Docker and Kubernetes (via Helm) to orchestrate the LiteLLM proxy, its database, and monitoring components. Hardening is achieved through non-root container environments, read-only filesystems, and restricted network egress, ensuring that the application remains secure even in high-stakes environments.
The deployment architecture is designed to support scalable LLM gateway services, integrating authentication, load balancing, and health monitoring. By utilizing specific Dockerfiles like Dockerfile.non_root, the project minimizes the attack surface of the running containers.
Sources: docker/README.md:1-10, AGENTS.md:14-25
Deployment Architecture
The standard LiteLLM deployment consists of a multi-container stack orchestrated via Docker Compose or Kubernetes. The primary service is the LiteLLM proxy, which interacts with a PostgreSQL database and a Prometheus instance for metrics.
The diagram shows the standard flow of an API request through the LiteLLM proxy and its dependencies.
Sources: docker/README.md:21-28, deploy/charts/litellm-helm/templates/deployment.yaml:85-110
Core Components and Services
| Component | Description | Default Port |
|---|---|---|
litellm | The main proxy application serving LLM requests. | 4000 |
litellm_db | PostgreSQL database for storing users, keys, and budgets. | 5432 |
prometheus | Monitoring service for scraping proxy metrics. | 9090 |
redis | Optional service for distributed rate limiting. | 6379 |
Sources: docker/README.md:21-28, deploy/charts/litellm-helm/templates/deployment.yaml:128-135
Hardening Strategies
Hardening is a critical aspect of LiteLLM deployment, specifically designed to meet enterprise security requirements such as restricted egress and read-only root filesystems.
Non-Root and Read-Only Filesystem
Production environments use Dockerfile.non_root, which ensures the application does not run with administrative privileges. The hardened configuration mounts the root filesystem as read-only, using tmpfs for specific writable paths needed for caching and migrations.
This diagram illustrates the separation between the read-only core and the limited writable mounts in a hardened environment.
Sources: docker/README.md:46-55, deploy/charts/litellm-helm/templates/deployment.yaml:194-200
Network and Egress Control
Hardened deployments route outbound traffic through a local Squid proxy that denies egress by default. This forces the application to rely on pre-baked assets. For example, Prisma migrations must use cached CLI engines instead of attempting to download them at runtime.
Sources: docker/README.md:58-62
Pre-built Assets
To support offline and hardened environments, the Admin UI and its assets are pre-built and served from specific read-only paths:
/var/lib/litellm/ui: Contains the pre-restructured Next.js UI./var/lib/litellm/assets: Contains UI logos and assets.- A
.litellm_ui_readymarker is used to signify a completed build.
Sources: docker/README.md:55-57
Environment Configuration
The LiteLLM container requires several environment variables for secure operation, most notably the MASTER_KEY used for token signing.
Required and Optional Variables
| Variable | Purpose | Default |
|---|---|---|
MASTER_KEY | Secret key for signing and validating tokens. | Required |
DATABASE_URL | Connection string for PostgreSQL. | Optional |
REDIS_HOST | Host for Redis caching/rate limiting. | Optional |
LITELLM_MIGRATION_DIR | Directory for Prisma migrations. | /app/migrations |
SEPARATE_HEALTH_APP | Enables a secondary app for health checks. | 0 |
Sources: docker/README.md:14-20, deploy/charts/litellm-helm/templates/deployment.yaml:65-125
Verification Procedures
Before full deployment, operators should verify the container's integrity and offline capabilities:
docker run --rm --network none --entrypoint prisma ghcr.io/berriai/litellm:main-stable --version
This command confirms that Prisma engines are available within the image without network access.
Sources: docker/README.md:65-68
Health Monitoring and Probes
LiteLLM supports standard Kubernetes probes (Liveness, Readiness, Startup) to ensure service availability. A dedicated Health Check Client is also provided for comprehensive model testing.
Kubernetes Probes
| Probe Type | Default Path | Default Failure Threshold |
|---|---|---|
| Liveness | /health/liveliness | 3 |
| Readiness | /health/readiness | 3 |
| Startup | /health/readiness | 30 |
Sources: deploy/charts/litellm-helm/templates/deployment.yaml:141-165, deploy/charts/litellm-helm/tests/deployment_tests.yaml:230-265
Health Check Client
The Health Check tool (containerized as litellm-health-check) queries configured models to verify if they are functional. It supports concurrent testing of chat and embedding models.
Sources: scripts/health_check/health_check_client_README.md:1-15
The sequence shows how the Health Check Client verifies the status of all configured models through the proxy.
Sources: scripts/health_check/health_check_client_README.md:85-95
Conclusion
Docker deployment and hardening in LiteLLM ensure that the proxy can be deployed in diverse environments, ranging from local developer setups to highly restricted enterprise clusters. By combining standard container orchestration with specialized security measures like non-root users, read-only filesystems, and offline Prisma support, LiteLLM maintains a secure and reliable gateway for Large Language Model integration.