Routing and Load Balancing
Relevant source files
The following files were used as context for generating this wiki page:
- litellm/router.py
- litellm/router_strategy/lowest_latency.py
- litellm/router_strategy/lowest_cost.py
- litellm/router_utils/fallback_event_handlers.py
- AGENTS.md
- ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx
- ui/litellm-dashboard/src/components/add_model/RouterConfigBuilder.test.tsx
Introduction
The Routing and Load Balancing system in LiteLLM serves as a unified orchestration layer designed to manage requests across multiple LLM deployments. It provides consistent OpenAI-format output regardless of the underlying provider while implementing sophisticated logic for high availability, cost optimization, and performance. This system allows developers to define a model_list and apply various strategies to determine which specific deployment should handle a given request.
The core of this system is the Router class, which handles lifecycle events, manages deployment health, and executes fallback logic when primary deployments fail. By abstracting the complexities of multi-provider management, the system ensures reliability through automated retries and load distribution across diverse regions and service tiers.
Sources: AGENTS.md:5-10, litellm/router.py:1-50
Core Components and Architecture
The Router Class
The Router is the primary orchestrator that accepts a model_list containing various LLM deployments (e.g., OpenAI, Azure, Anthropic). It is responsible for initializing deployments, tracking their health status, and routing calls based on predefined strategies.
The diagram above illustrates the high-level request flow through the Router, including the selection process and error handling loop.
Sources: litellm/router.py:1-100, AGENTS.md:12-18
Routing Strategies
LiteLLM supports several strategies to optimize for different business requirements. These strategies are implemented as specialized modules that calculate the "best" deployment based on real-time metrics.
| Strategy | Description | Key Metric |
|---|---|---|
| Lowest Latency | Selects the deployment with the fastest response time. | Response Time (ms) |
| Lowest Cost | Prioritizes deployments with the lowest input/output token pricing. | USD per 1M Tokens |
| Complexity Based | Routes based on the semantic complexity of the prompt. | Calculated Complexity Score |
| Basic Load Balancing | Distributes requests evenly across available healthy deployments. | Health Status & RPM |
Sources: litellm/router_strategy/lowest_latency.py, litellm/router_strategy/lowest_cost.py, ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx:41-47
Semantic Complexity Routing
The Complexity Router provides a rule-based scoring mechanism that classifies requests into tiers without incurring the latency of an additional LLM call. It evaluates requests across multiple dimensions to ensure that expensive, high-reasoning models are only used when necessary.
Scoring Dimensions
The weighted score (ranging from 0.0 to 1.0) is determined by evaluating:
- Token Count: Length of the input prompt.
- Code Presence: Detection of code blocks or programming-related syntax.
- Reasoning Markers: Presence of phrases like "think step by step."
- Technical Terms: Identification of specialized vocabulary.
- Multi-step Patterns: Indicators of complex instruction sets.
Complexity Tiers
| Tier | Score Range | Description |
|---|---|---|
| SIMPLE | < 0.15 | Basic questions, greetings, or factual queries. |
| MEDIUM | 0.15 - 0.35 | Standard queries requiring basic reasoning or explanation. |
| COMPLEX | 0.35 - 0.60 | Technical, multi-part requests requiring deep knowledge. |
| REASONING | > 0.60 | Chain-of-thought analysis or explicit reasoning requests. |
Sources: ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx:28-39, ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx:103-125
Error Handling and Fallbacks
Reliability is managed through fallback_event_handlers, which intercept provider-specific exceptions and map them to OpenAI-compatible errors. This triggers the Router's internal retry and deployment-switching mechanisms.
This sequence shows the automatic transition from a failing deployment to a healthy fallback deployment when a rate limit is encountered.
Sources: litellm/router_utils/fallback_event_handlers.py, AGENTS.md:95-103
Technical Implementation Details
Configuration Schema
The RouterConfigBuilder and associated UI components define the structure for configuring routing rules. A typical configuration includes a name, semantic utterances for training/matching, a description, and a score threshold.
{
"routes": [
{
"name": "gpt-4-route",
"utterances": ["complex analysis", "design architecture"],
"description": "Used for high-complexity tasks",
"score_threshold": 0.7
}
]
}
Sources: ui/litellm-dashboard/src/components/add_model/RouterConfigBuilder.test.tsx:41-52, ui/litellm-dashboard/src/components/add_model/RouterConfigBuilder.test.tsx:139-150
Latency and Cost Tracking
Strategies like lowest_latency and lowest_cost rely on persistent tracking of deployment performance.
- Lowest Latency: Periodically pings deployments or uses historical data to maintain a latency map.
- Lowest Cost: Uses the
model_prices_and_context_window.jsondatabase to calculate real-time cost per request based on token usage.
Sources: litellm/router_strategy/lowest_latency.py, litellm/router_strategy/lowest_cost.py, AGENTS.md:120-125
Conclusion
The Routing and Load Balancing system in LiteLLM provides a robust framework for managing multi-LLM environments. By combining performance-aware strategies (latency/cost) with semantic awareness (complexity tiers), the system optimizes for both cost and user experience. The integrated fallback handlers and health tracking ensure that the system remains resilient even when individual provider endpoints encounter downtime or rate limits.