Ordered Upstreams & Failover

Relevant source files

The following files were used as context for generating this wiki page:

• src/proxy.rs
• src/routing.rs
• src/config.rs
• ARCHITECTURE.md
• blueprints/upstream/_generic.md
• src/blueprints.rs

Ordered Upstreams and Failover represent the core routing logic of shunt, enabling the transparent redirection of Anthropic Messages requests to various providers based on model identifiers. The system ensures high availability and flexibility by resolving a request's model ID to a specific Route, which determines the upstream provider and protocol adapter used for inference.

The mechanism operates by matching requested model IDs against a hierarchy of configuration rules: exact routes, prefix-based routes, and finally a global default provider. This allows operators to divert specific traffic (e.g., for cost or performance reasons) while passing unmapped traffic through to Anthropic unchanged.

Sources: ARCHITECTURE.md:10-18, src/routing.rs

Routing Architecture

The routing system resolves a public model ID into a Route object. This resolution process is hierarchical and follows a strict order of precedence to ensure predictable behavior.

Resolution Hierarchy

1. Exact Routes: Matches the model ID exactly against the [[routes]] configuration.
2. Prefix Routes: Matches the start of the model ID against [route_prefixes].
3. Default Provider: If no specific route is found, the system falls back to the default_provider (which defaults to anthropic).

The following diagram illustrates the logical flow of model resolution during a request:

The routing logic also handles specific suffixes, such as stripping the [1m] context-window suffix from model IDs before resolution.

Sources: src/routing.rs, ARCHITECTURE.md:46-48, src/proxy.rs:20-25

Configuration and Upstreams

Upstreams are defined as provider endpoints that shunt can communicate with. These are configured using the [[upstreams]] table, where each entry specifies a name, provider type, base URL, and authentication method.

Upstream Components

Component	Description
name	A stable identifier for the upstream, used in routing maps.
kind	The protocol adapter type (e.g., anthropic, responses, cursor).
base_url	The API root for the upstream provider.
auth	Configuration for credentials (API keys, OAuth, or passthrough).

Sources: blueprints/upstream/_generic.md:31-40, src/blueprints.rs:48-60

Mapping Models to Upstreams

Models are mapped to upstreams via the [[models]] or legacy [[routes]] configuration. A model entry defines an id and an upstream_model map, which allows the gateway to translate the public ID into a provider-specific ID.

[[models]]
id = "claude-custom-model"
display_name = "Custom model"

[models.upstream_model]
custom_upstream = "provider-model-id"

Sources: blueprints/upstream/_generic.md:49-57

Failover and Error Handling

shunt prioritizes technical accuracy and streaming integrity during failover and error scenarios. Errors generated by the gateway itself are reshaped into the Anthropic error envelope to ensure compatibility with Claude Code.

Error Classification

• ShuntError: Represents gateway-level issues like invalid_request_error or authentication_error.
• UpstreamError: Represents network or transport failures (e.g., 502 Bad Gateway) when an upstream is unreachable or returns a malformed response.

Streaming and Failover

A critical invariant of the system is that streaming is never buffered. Upstream Server-Sent Events (SSE) are forwarded to the client immediately. Because of this, once a stream has started, the gateway cannot transparently fail over to a different upstream for the same request without breaking the client's incremental rendering. Failover typically occurs at the routing stage before the request is dispatched to an adapter.

Summary

The Ordered Upstreams and Failover system provides a robust mechanism for multi-provider routing within shunt. By leveraging a hierarchical resolution strategy and standardized protocol adapters, the gateway can effectively manage model traffic across diverse backends while maintaining strict adherence to the Anthropic Messages API. This ensures that developers can utilize alternative models without sacrificing the tools or streaming capabilities of the Claude Code environment.