Enterprise Deployment and Features
Relevant source files
The following files were used as context for generating this wiki page:
LiteLLM Enterprise provides a hardened, scalable version of the LLM Gateway designed for production environments. It extends the core capabilities of the open-source project by offering advanced management, enhanced security, and specialized deployment templates for cloud infrastructure. Sources: AGENTS.md, enterprise/README.md
The enterprise module includes specialized logic for handling complex requirements such as centralized authentication, granular rate limiting, and dedicated admin dashboards for team management. It is architected to integrate seamlessly with standard enterprise stacks, including Kubernetes (Helm) and AWS CloudFormation. Sources: AGENTS.md, enterprise/cloudformation_stack/litellm.yaml
Enterprise Architecture and Logic
The Enterprise system operates as a proxy server (LLM Gateway) that sits between client applications and 100+ LLM providers. Its primary responsibility is to standardize OpenAI-format outputs while enforcing enterprise-level policies. Sources: AGENTS.md
Data Flow and Management
The gateway processes requests by performing authentication, checking budgets/rate limits, and then routing to the appropriate deployment.
The Gateway utilizes a PostgreSQL database for persisting management data (keys, teams, models) and Redis for high-performance caching and rate limit tracking. Sources: AGENTS.md, deploy/charts/litellm-helm/templates/deployment.yaml:73-81
Key Components
| Component | Description |
|---|---|
| Proxy Server | Main FastAPI implementation handling request/response cycles. |
| Auth Module | Handles API key management, JWT, and OAuth2. |
| Management Endpoints | Admin APIs for managing keys, teams, and model groups. |
| Guardrails | Hooks for safety, PII masking, and content filtering. |
| Router | Load balancing logic across multiple deployments with fallback support. |
Sources: AGENTS.md
Cloud Infrastructure Deployment
LiteLLM Enterprise supports automated deployment through CloudFormation and Helm, ensuring consistent configuration for production traffic.
AWS CloudFormation Stack
The enterprise CloudFormation stack sets up a high-availability environment on AWS. It defines parameters for networking, compute, and database connectivity. Sources: enterprise/cloudformation_stack/litellm.yaml
The stack supports configuration of sensitive data, such as MasterKey, through AWS Secrets Manager and handles VPC networking requirements including Public and Private subnets. Sources: enterprise/cloudformation_stack/litellm.yaml:12-60
Kubernetes (Helm) Configuration
For Kubernetes environments, the deployment is managed via Helm charts which configure the container environment variables, readiness/liveness probes, and resource constraints. Sources: deploy/charts/litellm-helm/templates/deployment.yaml
# Example Environment Configuration from Helm
env:
- name: DATABASE_URL
value: {{ .Values.db.url | quote }}
- name: PROXY_MASTER_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.masterkeySecretName }}
key: masterkey
- name: REDIS_HOST
value: {{ include "litellm.redis.serviceName" . }}
Sources: deploy/charts/litellm-helm/templates/deployment.yaml:49-76
Enterprise Feature Set
The enterprise version includes specialized modules that are maintained separately from the open-source core. Sources: AGENTS.md, enterprise/README.md
Management and Security
- Centralized Auth: Dedicated logic in
litellm/proxy/auth/for enterprise identity providers. - Spend Tracking: Enhanced budgeting and rate limiting for multi-tenant environments.
- Audit Logs: Comprehensive logging for compliance and observability.
- Admin UI: A dashboard specifically built for managing enterprise entities like virtual keys and teams.
Deployment Parameters (Helm/CF)
| Parameter | Type | Description | Default |
|---|---|---|---|
DATABASE_URL | String | Connection string for PostgreSQL database. | - |
PROXY_MASTER_KEY | Secret | Master key for admin API access. | - |
REDIS_HOST | String | Hostname for Redis instance used for caching/rate limits. | - |
SEPARATE_HEALTH_APP | Boolean | Whether to run the health check app on a separate port. | 0 |
numWorkers | Integer | Number of Gunicorn/Uvicorn workers to spawn. | - |
Sources: deploy/charts/litellm-helm/templates/deployment.yaml:49-111, deploy/charts/litellm-helm/tests/deployment_tests.yaml
Conclusion
LiteLLM Enterprise is designed for high-throughput, secure LLM operations. By leveraging structured deployment templates (Helm/CloudFormation) and a robust proxy architecture, it allows enterprises to centralize LLM access while maintaining granular control over costs, security, and performance. Sources: enterprise/README.md, AGENTS.md