Budgets and Rate Limiting
Relevant source files
The following files were used as context for generating this wiki page:
The "Budgets and Rate Limiting" system within LiteLLM is a critical infrastructure component designed to manage resource allocation, ensure financial guardrails, and maintain system stability across a unified interface for 100+ LLM providers. Its primary purpose is to enforce spending limits (budgets) and throughput constraints (rate limits) at multiple levels, including users, API keys, teams, and specific models.
This system operates primarily through a series of "Proxy Hooks" that intercept incoming requests before they reach the LLM providers. By integrating with Redis for distributed state management and PostgreSQL for persistent configuration, the system provides high-performance, real-time enforcement of complex allocation policies such as saturation-aware priority rate limiting and project-wide spend tracking.
Sources: ARCHITECTURE.md, litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:3-8
Budget Management Architecture
Budget management in LiteLLM is handled by the BudgetManager class and specialized proxy hooks. It tracks cumulative spending against pre-defined limits.
Core Components
- BudgetManager: A core class that manages user costs and project budgets. It supports both local JSON-based storage and hosted database backends.
- MaxBudgetLimiter Hook: A specialized proxy hook (
MaxBudgetLimiter) that enforces maximum budget constraints during the pre-call phase of an API request. - Persistent Storage: Utilizes Redis for fast spend increments and PostgreSQL for long-term spend logs and configuration.
Sources: litellm/budget_manager.py:27-37, litellm/proxy/hooks/max_budget_limiter.py:17-25, ARCHITECTURE.md
Budget Tracking Logic
The system calculates the cost of a request based on token usage and provider-specific pricing. It supports multiple reset durations: daily, weekly, monthly, and yearly.
The diagram represents the flow from request ingestion to persistent spend logging.
Sources: litellm/budget_manager.py:68-76, ARCHITECTURE.md, litellm/proxy/hooks/max_budget_limiter.py:112-120
Budget Reset Intervals
Resetting logic is based on the budget_duration configuration.
| Duration | Implementation Detail |
|---|---|
| Daily | Resets after 24 hours |
| Weekly | Resets after 7 days |
| Monthly | Resets after 30 days |
| Yearly | Resets after 365 days |
Sources: litellm/budget_manager.py:148-164
Rate Limiting Systems
LiteLLM employs multiple rate limiting strategies to prevent system over-saturation and ensure fair resource distribution.
Parallel Request Limiter (v3)
The _PROXY_ParallelRequestLimiter manages concurrent requests. It uses Redis counters to track active requests per API key or user. If the number of active requests exceeds the max_parallel_requests threshold, the request is throttled with a 429 status code.
Sources: litellm/proxy/hooks/parallel_request_limiter_v3.py:17-40
Dynamic Rate Limiting (v3)
The v3 dynamic rate limiter is "saturation-aware." It adjusts its behavior based on the current load of the system relative to total capacity.
Operating Modes
- Generous Mode (Saturation < 80%): Allows users to "borrow" unused capacity from other priority tiers. It focuses on maximizing throughput.
- Strict Mode (Saturation >= 80%): Enforces strict priority-based limits and normalizes weights to ensure fairness during high load.
Sources: litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:10-15, litellm/proxy/hooks/dynamic_rate_limiter.py:46-55
Priority Reservation Flow
Priority-based allocation uses weights (e.g., premium: 0.75, standard: 0.25) to distribute model capacity.
The sequence diagram illustrates the decision logic for selecting the rate limiting enforcement mode based on Redis-queried saturation.
Sources: litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:20-80, litellm/proxy/hooks/dynamic_rate_limiter.py:102-125
Configuration Elements
The system is configured through environment variables and a YAML configuration file for the proxy.
Key Configuration Parameters
| Parameter | Type | Description |
|---|---|---|
max_budget | float | Total allowed spend in USD. |
budget_duration | string | Interval for budget resets (e.g., "30d"). |
rpm_limit | int | Requests Per Minute limit. |
tpm_limit | int | Tokens Per Minute limit. |
priority_reservation | dict | Map of priority labels to capacity weights (0.0 - 1.0). |
Sources: litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:85-100, litellm/budget_manager.py:84-90, ARCHITECTURE.md
Priority Reservation Settings
Developers can fine-tune the saturation behavior using the PriorityReservationSettings object.
# Example Configuration
litellm.priority_reservation_settings = PriorityReservationSettings(
default_priority=0.5, # Weight for users without metadata
saturation_threshold=0.80, # Threshold for strict enforcement
tracking_multiplier=10 # Multiplier for non-blocking tracking
)
Sources: litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:103-115
Implementation Details
Redis Counter Logic
The system uses Redis as a high-speed, distributed counter. Rate limit descriptors are created dynamically based on the request context (model, user, or key). In strict mode, these descriptors are normalized to prevent the sum of allocated capacity from exceeding 100% of the provider's limit.
Sources: litellm/proxy/hooks/dynamic_rate_limiter.py:165-185, litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:120-130
Failure Handling
If the budget or rate limit is exceeded, the hooks raise specific exceptions:
- Budget Exceeded: Returns a 400 Bad Request with a message identifying the exhausted budget.
- Rate Limit Exceeded: Returns a 429 Too Many Requests, often accompanied by
retry-afterheaders if supported by the provider.
Sources: litellm/proxy/hooks/max_budget_limiter.py:112-120, litellm/proxy/hooks/parallel_request_limiter_v3.py:133-145
Summary
LiteLLM provides a robust, multi-layered system for managing budgets and rate limits. By combining static budget tracking with dynamic, saturation-aware rate limiting, the project ensures that resource consumption remains within project boundaries while providing flexibility for high-priority traffic during peak demand periods. This architecture is central to the "AI Gateway" functionality, allowing for enterprise-grade control over disparate LLM provider usage.
Sources: ARCHITECTURE.md, litellm/proxy/hooks/README.dynamic_rate_limiter_v3.md:3-15