Background Jobs and Schedulers
Relevant source files
The following files were used as context for generating this wiki page:
Background jobs and schedulers in LiteLLM manage asynchronous tasks essential for maintaining the proxy's operational integrity, such as budget management, security through key rotations, and persistent data synchronization. These jobs operate outside the main request-response loop to ensure high performance and reliability for end-users.
The system relies on a combination of periodic loops, scheduled managers, and persistent writers that interface with databases (PostgreSQL/SQLite) and caches (Redis). These components handle critical administrative tasks like resetting user budgets, rotating virtual keys, and updating spend logs.
Sources: AGENTS.md, litellm/proxy/proxy_server.py
Core Background Systems
Budget Reset Job
The Budget Reset Job is responsible for periodically clearing the spent budget for users, keys, and teams based on their defined reset intervals (e.g., daily, monthly). This ensures that rate limits and financial constraints are applied correctly over time.
The diagram above shows the logic flow for resetting budgets within the system.
Sources: litellm/proxy/common_utils/reset_budget_job.py
Key Rotation Manager
The KeyRotationManager handles the lifecycle of virtual keys, ensuring that keys are rotated according to security policies. It monitors key age and usage, triggering rotations when specific criteria are met to minimize the impact of potential key leakage.
Key features include:
- Scheduled Rotations: Automatically generates new keys and retires old ones.
- Grace Periods: Ensures connectivity isn't lost during the switchover.
Sources: litellm/proxy/common_utils/key_rotation_manager.py, cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md
Database Spend Update Writer
The DBSpendUpdateWriter is a persistent background worker that batches spend updates from memory or cache and writes them to the primary database. This prevents database bottlenecks during high-concurrency LLM calls.
The sequence diagram illustrates how spend data is asynchronously synchronized between the high-speed cache and persistent storage.
Sources: litellm/proxy/db/db_spend_update_writer.py
Infrastructure and Configuration
Background jobs are typically configured via environment variables or the proxy_config.yaml. In distributed deployments (e.g., Kubernetes via Helm), these jobs may be controlled by replica settings or separate health checks to ensure at least one instance is performing maintenance tasks.
| Component | Configuration Method | Default Interval | Description |
|---|---|---|---|
| Budget Reset | proxy_config.yaml | Periodic | Resets spend for entities with reset_interval. |
| Spend Writer | Environment Variables | Continuous | Batches and writes spend data to DB. |
| Key Rotation | Admin UI / Config | Scheduled | Manages automatic virtual key updates. |
Sources: deploy/charts/litellm-helm/templates/deployment.yaml, litellm/proxy/proxy_server.py
Deployment Considerations
In containerized environments, background jobs are often managed within the main proxy container but can be influenced by the following settings:
replicaCount: In the Helm chart, this determines how many instances of the proxy (and its background jobs) are running.terminationGracePeriodSeconds: Defaults to 90 seconds to allow background writers to finish flushing data to the database before the pod terminates.
Sources: deploy/charts/litellm-helm/templates/deployment.yaml:31, deploy/charts/litellm-helm/templates/deployment.yaml:239
Conclusion
Background jobs and schedulers are the backbone of LiteLLM's management layer. By decoupling spend updates, budget resets, and key rotations from the main execution path, the system achieves a balance between strict administrative control and low-latency response times for LLM consumers.