Proxy Request Flow

Relevant source files

The following files were used as context for generating this wiki page:

• ARCHITECTURE.md
• AGENTS.md
• litellm/proxy/common_request_processing.py
• litellm/proxy/route_llm_request.py
• litellm/llms/litellm_proxy/skills/README.md

The Proxy Request Flow in LiteLLM represents the end-to-end lifecycle of an LLM request as it passes through the AI Gateway. The gateway acts as a unified interface, wrapping the LiteLLM SDK with enterprise-grade features including authentication, rate limiting, budget management, and intelligent routing across 100+ LLM providers.

Sources: ARCHITECTURE.md, AGENTS.md

High-Level Request Lifecycle

A request originating from a client (e.g., OpenAI SDK, Anthropic SDK, or generic HTTP client) is processed by the Proxy Server before being handed off to the internal SDK for provider-specific translation and execution.

The following sequence diagram illustrates the complete path from the initial client POST request to the final response and asynchronous cost logging.

Proxy Components and Orchestration

The AI Gateway architecture is modular, separating the entry point logic from internal utilities and persistent infrastructure.

Core Architectural Layers

The request flow moves through several distinct layers within the litellm/proxy/ directory:

1. Entry Point: proxy_server.py defines the FastAPI application and main API endpoints.
2. Authentication: proxy/auth/ handles API key validation, JWT parsing, and OAuth2 flows.
3. Pre-Call Logic: litellm_pre_call_utils.py and common_request_processing.py prepare the request metadata and extract costs.
4. Routing: route_llm_request.py determines which deployment or model group should handle the request.
5. Hooks: proxy/hooks/ implements middleware-like callbacks for budget enforcement and rate limiting.

Sources: ARCHITECTURE.md:70-109, litellm/proxy/common_request_processing.py

Infrastructure Integration

The gateway utilizes a dual-cache system and a relational database to ensure low-latency request processing and reliable spend tracking.

Specialized Flow: Skills and Code Execution

LiteLLM supports database-backed "Skills" that inject logic or tools into the request flow. This is particularly relevant for non-Anthropic providers that do not natively support the Skills API.

Skills Injection Hook

When a request contains container.skills, the SkillsInjectionHook intercepts the flow:

• Skill Retrieval: Fetches the skill (e.g., litellm:skill_abc) from the LiteLLM database.
• System Prompt Injection: Extracts SKILL.md from the stored ZIP file and appends it to the system prompt.
• Tool Conversion: Converts the skill into an OpenAI-style function tool.

Agentic Loop for Code Execution

If a skill requires Python code execution, the proxy manages an internal loop using a Docker sandbox.

Phase	Action
Pre-call	Injects litellm_code_execution tool and SKILL.md content.
Response Detection	Identifies if the model called the code execution tool.
Execution	Runs the Python code in a Docker sandbox.
Loop Continuation	Appends tool results to messages and re-calls the LLM until completion.

Sources: litellm/llms/litellm_proxy/skills/README.md:1-85

Cost Attribution and Background Jobs

A critical part of the proxy request flow is the asynchronous processing of usage data to prevent blocking the client response.

1. Calculation: After the SDK call, litellm.completion_cost() calculates the cost based on tokens and provider pricing.
2. Header Injection: common_request_processing.py extracts this cost and adds the x-litellm-response-cost header to the HTTP response.
3. Spend Queuing: DBSpendUpdateWriter queues the spend increment in Redis.
4. Persistence: A background job (update_spend) flushes these increments to PostgreSQL every 60 seconds.

Sources: ARCHITECTURE.md:162-177, ARCHITECTURE.md:195-212

Summary

The LiteLLM Proxy Request Flow provides a robust wrapper around standard LLM interactions. By intercepting requests at the gateway level, it enables centralized management of security, costs, and routing while maintaining compatibility with major provider SDKs. The inclusion of advanced features like skills injection and automated code execution further extends the proxy's capabilities into agentic workflows.

Sources: ARCHITECTURE.md, AGENTS.md, litellm/llms/litellm_proxy/skills/README.md