Introduction to LiteLLM
Relevant source files
The following files were used as context for generating this wiki page:
LiteLLM is a unified interface designed to streamline interactions with over 100 Large Language Model (LLM) providers. It acts as a translation layer that normalizes inputs and outputs, allowing developers to interact with diverse models—such as those from OpenAI, Anthropic, Azure, and Google—using a consistent OpenAI-compatible format.
The project provides a core library for programmatic integration and a high-performance proxy server (LLM Gateway). Key features include standardized tool calling, load balancing across multiple deployments, retry/fallback logic, and robust observability integrations.
Sources: AGENTS.md:7-13, CLAUDE.md:13-14
Core Architecture
LiteLLM is structured into two primary components: the Core Library and the Proxy Server.
Core Library (litellm/)
The core library contains the logic for translating generic requests into provider-specific API calls.
- Main Entry Point: The
completion()function inlitellm/main.pyhandles the primary text generation requests. - Provider Implementations: Located in
litellm/llms/, these subdirectories contain the transformation logic for each specific LLM provider. - Router System: Managed by
litellm/router.py, it provides load balancing and fallback mechanisms across different model deployments. - Type Definitions: Pydantic models in
litellm/types/ensure data validation and type safety across the library.
Sources: CLAUDE.md:16-22, AGENTS.md:17-23
Proxy Server (litellm/proxy/)
The Proxy Server is a FastAPI-based LLM Gateway that extends the core library's functionality into a standalone service.
- Authentication: Supports API keys, JWT, and OAuth2 for secure access.
- Management Endpoints: Admin APIs for managing virtual keys, teams, and model configurations.
- Database: Utilizes Prisma ORM with PostgreSQL or SQLite support for storing configurations and usage data.
- Observability: Built-in hooks for logging, spend tracking, and guardrails.
Sources: CLAUDE.md:24-31, AGENTS.md:104-112
System Flow Diagram
The following diagram illustrates the flow of a request through the LiteLLM architecture:
The diagram shows how both SDK users and HTTP clients reach the translation layer, which then routes requests to specific provider APIs and returns a standardized response.
Sources: CLAUDE.md:16-31, AGENTS.md:8-13
Key Functional Components
Provider Implementation Pattern
Each provider implementation follows a consistent pattern to ensure compatibility. Providers inherit from base classes and implement transformation functions for request and response formatting. This allows LiteLLM to support advanced features like streaming and tool calling across disparate APIs.
Sources: CLAUDE.md:35-39, AGENTS.md:37-43
Router and Load Balancing
The Router component manages request distribution across multiple deployments of the same or different models. It implements:
- Load Balancing: Distributing traffic to prevent overloading a single endpoint.
- Fallbacks: Automatically retrying a request with a secondary provider if the primary fails.
- Retry Logic: Handling transient errors based on configurable policies.
Sources: AGENTS.md:11, CLAUDE.md:19
Database Schema (Prisma)
The proxy server uses a relational database to manage its state. One specific table, the LiteLLM_SkillsTable, demonstrates the structure used for advanced features:
| Field | Type | Description |
|---|---|---|
skill_id | String (UUID) | Primary key for the skill |
display_title | String | User-facing name of the skill |
file_content | Bytes | ZIP file containing skill logic and metadata |
created_at | DateTime | Timestamp of creation |
metadata | Json | Extensible metadata for the record |
Sources: litellm/llms/litellm_proxy/skills/README.md:162-177
Development Guidelines
Code Quality Standards
LiteLLM enforces strict quality standards to maintain its extensive provider support:
- Formatting: Uses Black for code formatting.
- Linting: Uses Ruff for fast linting and code quality checks.
- Type Checking: Employs MyPy for static type validation.
- Validation: Leverages Pydantic v2 for data validation.
Sources: CONTRIBUTING.md:144-150, CLAUDE.md:46-48
Testing Strategy
The project maintains a multi-tiered testing approach:
- Unit Tests: Located in
tests/test_litellm/, these focus on logic without external API calls (mocked). - Integration Tests: Found in
tests/llm_translation/, these test against real provider APIs. - Proxy Tests: Validates gateway-specific features like authentication and routing.
Sources: CLAUDE.md:52-56, CONTRIBUTING.md:73-77
Summary
LiteLLM provides a robust framework for abstracting LLM complexities through its unified translation layer. By separating the core translation logic from the management and security features of the Proxy Server, it offers a scalable solution for both individual developers and enterprise environments requiring centralized LLM governance.
Sources: AGENTS.md:7-13, CLAUDE.md:13-14