Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
LiteLLM is designed as a unified interface and gateway for over 100 LLM providers. It translates standardized inputs into provider-specific endpoints for completion, embedding, and image generation, while returning a consistent OpenAI-compatible format. The system is divided into two primary logical components: the Core Library, which handles low-level translation and routing, and the Proxy Server (LLM Gateway), which provides operational features like authentication, rate limiting, and management APIs.
Sources: AGENTS.md:5-12, CLAUDE.md:43-45, GEMINI.md:41-43
Core Library Architecture
The Core Library serves as the engine for request transformation and routing. It encapsulates the complexity of multiple API protocols into a single interface.
Component Breakdown
- Main Entry Point: The
completion()function inlitellm/main.pyis the primary interface for users. - Provider Implementations: Located in
litellm/llms/, these modules contain transformation logic for specific providers (e.g., OpenAI, Anthropic, Bedrock). - Router System: Managed via
litellm/router.py, this handles load balancing across multiple deployments and provides fallback logic. - Type Definitions: Pydantic models in
litellm/types/ensure data validation and type safety across the library.
Sources: CLAUDE.md:46-52, GEMINI.md:44-50, AGENTS.md:15-20
Request Flow
The following diagram illustrates how a request is processed through the Core Library.
The Core Library translates inputs to provider-specific endpoints and standardizes the output.
Sources: AGENTS.md:6-12, CLAUDE.md:55-58
Proxy Server (LLM Gateway)
The Proxy Server is a FastAPI-based application that wraps the core library with production-grade gateway features.
Key Management and Security
- Authentication: Supports API keys, JWT, and OAuth2.
- Database: Uses Prisma ORM with support for PostgreSQL and SQLite to store keys, teams, and usage data.
- Management Endpoints: Admin APIs located in
management_endpoints/allow for the programmatic creation of keys and team management.
Sources: CLAUDE.md:54-62, GEMINI.md:52-60, AGENTS.md:112-118
Proxy Server Components
| Component | Functionality |
|---|---|
| FastAPI App | Main server entry point in proxy_server.py. |
| Policy Engine | Combines guardrails with scoping rules (teams, keys, models). |
| Guardrails | Content filtering and safety hooks. |
| Pass-through | Forwards provider-specific unique endpoints directly. |
| UI Dashboard | A Next.js based admin interface for visual management. |
Sources: CLAUDE.md:54-63, litellm/proxy/policy_engine/architecture.md:5-9
Policy Engine Architecture
The Policy Engine allows administrators to define scoping rules for guardrails. It supports inheritance, allowing base policies to be extended by specific team or model policies.
Execution Flow
- Startup: Policies are loaded from
config.yamlinto an in-memoryPolicyRegistry. - Request Matching: The
PolicyMatcherchecks the incomingteam_alias,key_alias, andmodelagainst registered policies. - Resolution: The
PolicyResolvertraverses the inheritance chain to determine the final set of guardrails to apply.
The Policy Engine matches request context against defined scopes to apply appropriate guardrails.
Sources: litellm/proxy/policy_engine/architecture.md:11-25
UI Architecture (LiteLLM Dashboard)
The dashboard is built using Next.js and follows a strict file-based routing structure.
Structure and Constraints
- Routing: Folder-based routing where parenthesis (e.g.,
(dashboard)) are used to group logic without affecting the URL path. - Component Design: Components are intended to be "dumb," focusing only on rendering data provided via props or hooks. Files exceeding 300 lines are required to be broken down.
- Common Logic: Components and hooks used across multiple pages are moved to the lowest common ancestor directory.
Sources: ui/litellm-dashboard/src/app/(dashboard)/README.md:5-25
Common Implementation Patterns
Provider Transformations
Providers inherit from base classes in litellm/llms/base.py. Each implementation must provide transformation functions to handle:
- Input mapping (OpenAI params to Provider params).
- Output standardization (Provider response to OpenAI-like object).
- Error mapping (Provider exceptions to standard error codes).
Sources: CLAUDE.md:65-70, GEMINI.md:63-68, AGENTS.md:25-30
ChatGPT/Codex OAuth Logic
Specialized utilities in litellm/llms/chatgpt/common_utils.py manage OAuth tokens and device-based authentication for specific subscription-based flows. It includes logic for generating terminal-specific User-Agents and managing session IDs.
Sources: litellm/llms/chatgpt/common_utils.py:10-30, litellm/llms/chatgpt/common_utils.py:136-160