Pass-Through Endpoints & Policy Engine

Relevant source files

The following files were used as context for generating this wiki page:

• litellm/proxy/policy_engine/architecture.md
• ARCHITECTURE.md
• AGENTS.md
• CLAUDE.md
• GEMINI.md
• tests/llm_translation/test_bedrock_completion.py

Introduction

The Pass-Through Endpoints and Policy Engine represent critical components of the LiteLLM AI Gateway architecture. Pass-through endpoints allow the proxy to forward requests directly to provider-specific APIs while maintaining the unified monitoring and management features of the gateway. This enables support for specialized provider features that may not yet be fully abstracted by the core SDK.

The Policy Engine serves as the administrative control center, allowing for the definition of rules that combine guardrails with scoping logic. It targets specific teams, API keys, and models to ensure that incoming requests adhere to organizational safety and budget requirements before being processed by the underlying LLM providers.

Sources: ARCHITECTURE.md, litellm/proxy/policy_engine/architecture.md:3-8

Pass-Through Endpoints Architecture

Pass-through endpoints act as a direct bridge between the client and the LLM provider API, bypassing some of the standard OpenAI-format abstractions when necessary. This is particularly useful for provider-specific features like Vertex AI or Anthropic's native Messages API.

Component Relationship

The following diagram illustrates how pass-through requests flow through the proxy server to the provider APIs.

Key Pass-Through Endpoints

The gateway supports several specialized endpoints for direct provider interaction:

Endpoint	Purpose	Provider Context
/v1/messages	Anthropic native Messages API	Anthropic
/vertex-ai/*	Direct passthrough to Vertex AI	Google Cloud
/gemini/*	Passthrough to Google AI Studio	Google Gemini
/v1/rerank	Reranking endpoint support	Various
/* (General)	Direct passthrough for non-standard routes	All supported

Sources: ARCHITECTURE.md

Policy Engine Architecture

The Policy Engine operates as an interceptor within the request flow. It evaluates the context of an incoming request (team, key, model) against a set of registered policies to determine which guardrails or restrictions should be applied.

Architecture Overview

The engine consists of four primary internal components: the Policy Registry, Validator, Matcher, and Resolver.

Policy Engine Components

Component	Responsibility
PolicyRegistry	An in-memory singleton that acts as the primary store for all parsed and validated policies.
PolicyValidator	Responsible for checking configurations for errors in guardrail definitions, inheritance chains, and scope patterns.
PolicyMatcher	Compares the request context (metadata) against policy scopes using wildcard patterns for teams, keys, and models.
PolicyResolver	Processes the inheritance hierarchy, merging base policies and resolving additions or removals of guardrails.

Sources: litellm/proxy/policy_engine/architecture.md:32-37

Operational Flow

The interaction between these systems occurs in two distinct phases: system startup and per-request execution.

1. Initialization Phase

During proxy startup, the init_policies() function is triggered. It reads the policy configurations from the config.yaml file. These configurations are passed to the PolicyValidator to ensure there are no circular dependencies or invalid model patterns. Once validated, the policies are populated into the PolicyRegistry.

2. Request Execution Phase

When a request hits an endpoint (standard or pass-through):

1. Context Extraction: The gateway extracts the team_alias, key_alias, and model from the request.
2. Matching: The PolicyMatcher identifies all policies whose scope covers the current request context.
3. Resolution: The PolicyResolver takes the matching policies, follows any inheritance chains, and compiles the final list of guardrails to apply.
4. Application: The identified guardrails are then executed against the request before it is forwarded to the LLM provider.

Sources: litellm/proxy/policy_engine/architecture.md:39-44, ARCHITECTURE.md

Implementation Details

Pass-through logic often utilizes the allm_passthrough_route or llm_passthrough_route functions. For example, testing of Bedrock passthrough ensures that specific provider-specific headers (like anthropic_version) and data structures are preserved during the relay.

# Example of a passthrough request test
response = await litellm.allm_passthrough_route(
    model="bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0",
    method="POST",
    endpoint="/model/us.anthropic.claude-3-5-sonnet-20240620-v1:0/invoke",
    data=data,
)

Sources: tests/llm_translation/test_bedrock_completion.py:1571-1576

Conclusion

The Pass-Through Endpoints and Policy Engine provide the LiteLLM Proxy with the flexibility to support advanced provider features while maintaining strict administrative control. By separating policy registration, matching, and resolution, the system ensures that complex guardrail inheritance can be managed efficiently at scale across diverse teams and models.