Core SDK and Request Transformations
Relevant source files
The following files were used as context for generating this wiki page:
The LiteLLM Core SDK serves as a unified interface for over 100 LLM providers, standardizing inputs and outputs to match the OpenAI API format. It handles the complexity of provider-specific authentication, parameter mapping, and response parsing, allowing developers to switch between models like GPT-4, Claude 3, and Gemini with minimal code changes. The SDK manages both synchronous and asynchronous operations, streaming, and advanced features such as function calling and prompt caching.
At the heart of the SDK is a robust translation layer. This layer converts standardized OpenAI-format requests into the specific payloads required by providers such as AWS Bedrock, Anthropic, or Vertex AI. Conversely, it transforms raw provider responses back into standardized ModelResponse objects. This architecture ensures consistency across the entire ecosystem, whether using the SDK directly or through the LiteLLM AI Gateway.
Sources: ARCHITECTURE.md:1-10, AGENTS.md:7-12, ARCHITECTURE.md:162-175
SDK Request Flow and Orchestration
The SDK orchestrates LLM calls through a multi-staged pipeline that begins at entry points like litellm.completion() or litellm.acompletion(). These functions act as the primary interface for users, receiving parameters such as model names, messages, and optional settings.
Orchestration Components
The orchestration logic is distributed across several key files and classes:
main.py: Contains the primary entry points (completion,acompletion,embedding).utils.py: Provides theget_llm_provider()function, which resolves the model string (e.g.,bedrock/claude-3) to its corresponding LLM provider.BaseLLMHTTPHandler: Located inllms/custom_httpx/llm_http_handler.py, this class acts as the central orchestrator for HTTP-based providers, invoking transformation logic and handling the actual network requests.
Sources: ARCHITECTURE.md:162-185, litellm/main.py, litellm/utils.py
Request Flow Diagram
The following sequence diagram illustrates the flow of a request through the SDK components from the initial call to the final response callback.
The diagram shows how LiteLLM modularizes the request lifecycle, separating orchestration from provider-specific logic.
Sources: ARCHITECTURE.md:164-184
Translation Layer and Parameter Mapping
The translation layer is responsible for the interoperability between the standardized OpenAI format and various provider-specific formats. Each provider implementation includes a Config class (inheriting from BaseConfig) that defines how requests and responses are mapped.
Provider Transformation Logic
Each provider's transformation logic follows a specific contract:
transform_request(): Maps standard parameters liketemperature,max_tokens, andmessagesto the provider's specific fields. For example, AWS Bedrock's Converse API requires mappingmax_tokenstomaxTokenswithin aninferenceConfigblock.transform_response(): Parses the provider's unique JSON structure to extract generated text, usage statistics, and finish reasons, packaging them into aModelResponse.
Sources: ARCHITECTURE.md:200-215, tests/llm_translation/test_bedrock_completion.py:734-750
Parameter Support and Filtering
LiteLLM tracks which OpenAI parameters are supported by each provider to prevent errors during API calls. Unsupported parameters may be dropped or mapped to close equivalents depending on the configuration.
| OpenAI Parameter | Bedrock Converse Mapping | Gemini Mapping |
|---|---|---|
max_tokens | maxTokens (in inferenceConfig) | maxOutputTokens |
temperature | temperature | temperature |
top_p | topP | topP |
stop | stopSequences | stopSequences |
tools | toolConfig | tools |
Sources: tests/llm_translation/test_bedrock_completion.py:740-760, tests/llm_translation/test_openai.py:380-400
Specialized Transformations
LiteLLM handles complex data structures beyond simple text, including tool calls (function calling) and multimodal inputs (images and documents).
Tool and Function Calling
LiteLLM standardizes tool calling using the OpenAI format. When a model like Claude 3 on Bedrock is used, LiteLLM translates OpenAI tool definitions into Bedrock-specific toolSpec objects. It also handles the return trip, converting Bedrock's toolUse responses back into OpenAI-compatible tool_calls.
Sources: AGENTS.md:65-69, tests/llm_translation/test_bedrock_completion.py:465-485
Multimodal Transformation Flow
Multimodal inputs, such as images, require encoding and specific payload structures. LiteLLM handles the conversion of image URLs or base64 data into the format expected by the target provider.
This flowchart illustrates the internal branching logic for processing multimodal content blocks during request transformation.
Sources: tests/llm_translation/test_bedrock_completion.py:365-385, tests/llm_translation/test_openai.py:90-110
SDK Error Handling and Exceptions
LiteLLM maps provider-specific error codes (e.g., AWS Bedrock's ServiceUnavailableException or Anthropic's rate_limit_error) to a set of standardized OpenAI-compatible exceptions. This allows developers to use a single try-except block to catch errors regardless of the underlying provider.
Standardized Exception Mapping
| Provider Error | LiteLLM Standard Exception | Description |
|---|---|---|
| HTTP 429 | RateLimitError | Request limit exceeded. |
| HTTP 400 (Bad Format) | BadRequestError | Invalid parameters or payload. |
| HTTP 401/403 | AuthenticationError | Invalid API key or permissions. |
| HTTP 503 | ServiceUnavailableError | Provider service is down or overloaded. |
| Context Exceeded | ContextWindowExceededError | Input too long for the model's window. |
Sources: ARCHITECTURE.md:191-196, tests/llm_translation/test_bedrock_completion.py:700-715
Performance and Infrastructure Integration
The Core SDK integrates with infrastructure components like Redis for caching and cost tracking.
Caching and Cost Calculation
- Caching: The SDK supports response caching (e.g., via Redis) to reduce latency and costs for duplicate prompts. This is handled by the
LLMCachingHandler. - Cost Attribution: After a request completes,
utils.pyinvokes a cost calculator that uses token usage data from the response and pricing data from a internal map to determine the exact cost of the call.
Sources: ARCHITECTURE.md:120-145, ARCHITECTURE.md:150-160
Cost Tracking Architecture
The flow demonstrates how cost is calculated and attached to the response object before being passed to logging integrations.
Sources: ARCHITECTURE.md:150-160
Summary
Core SDK and Request Transformations represent the foundational intelligence of LiteLLM. By abstracting provider-specific APIs into a unified OpenAI-compatible interface, LiteLLM simplifies LLM integration while providing advanced orchestration, error mapping, and multimodal support. The translation layer's modular design ensures that adding or updating providers is isolated and testable, maintaining stability across the entire library.
Sources: ARCHITECTURE.md:200-220, AGENTS.md:1-15