Bounded Upstream Retry

Relevant source files

The following files were used as context for generating this wiki page:

• src/retry.rs
• tests/retry.rs
• ARCHITECTURE.md
• src/proxy.rs
• src/config.rs
• src/adapters/mod.rs

Bounded Upstream Retry is a resiliency mechanism within the shunt gateway designed to handle transient network failures or upstream provider errors before headers are sent to the client. Its primary goal is to improve the reliability of requests by automatically retrying failed operations within specific constraints, ensuring that the gateway maintains high availability even when upstream services experience intermittent issues.

The retry logic is strictly "bounded" to the pre-response phase. Once an upstream response begins streaming headers to the client, the gateway can no longer safely retry the request without violating streaming invariants or causing protocol errors. This mechanism is particularly critical for maintaining the Claude Code LLM gateway protocol compliance while supporting diverse backends.

Sources: ARCHITECTURE.md:46-49, src/retry.rs:1-10

Architecture and Core Logic

The retry system is implemented as a wrapper around asynchronous operations that return a Result. It uses a classification system to determine whether a specific error is "retryable" based on the state of the request and the nature of the failure.

Retry Classification

Errors are classified into three distinct categories to guide the retry loop:

• Retryable: The error is transient (e.g., a connection reset before headers) and a subsequent attempt may succeed.
• Permanent: The error is definitive (e.g., authentication failure or invalid request) and should be returned to the client immediately.
• Too Late: The failure occurred after the gateway committed to a response (e.g., mid-stream disconnect), meaning the state is now inconsistent for a retry.

Sources: src/retry.rs:35-45, src/adapters/mod.rs:55-60

Execution Flow

The following diagram illustrates the logical flow of a bounded retry attempt within the proxy layer:

Key Components

Functions and Data Structures

Component	Description	Location
send_with_retry	The primary orchestrator that wraps reqwest calls with retry logic and exponential backoff.	src/retry.rs
RetryClassifier	A trait or logic block used to determine if a reqwest::Error or AdapterError is transient.	src/retry.rs
AdapterFailure	An enum identifying the stage of failure (e.g., BeforeHeaders, UpstreamStatus).	src/adapters/mod.rs
max_retries	A configuration parameter defining the upper bound of retry attempts.	src/config.rs

Sources: src/retry.rs:15, src/adapters/mod.rs:65, src/config.rs:412

Classification Logic Implementation

The system distinguishes between transport-level errors and application-level errors. For example, a 502 Bad Gateway from an upstream load balancer is often considered retryable, whereas a 401 Unauthorized is treated as a permanent failure.

// Example of error classification logic derived from src/retry.rs
fn classify(err: &reqwest::Error) -> RetryAction {
    if err.is_connect() || err.is_timeout() {
        RetryAction::Retryable
    } else {
        RetryAction::Permanent
    }
}

Sources: src/retry.rs:50-65, tests/retry.rs:10-25

Constraints and Invariants

The "Before Headers" Bound

The gateway maintains a strict invariant: Streaming is never buffered. Because shunt prioritizes time-to-first-token and incremental rendering for Claude Code, retries are only possible if the failure happens before any bytes are written to the client response stream.

Configuration

Retries are globally configured within the server settings. This prevents infinite loops and resource exhaustion under sustained upstream outages.

Key	Type	Default	Description
server.max_retries	Integer	2	Number of times to attempt an upstream request before failing.
server.retry_delay_ms	Integer	100	Base delay for exponential backoff between attempts.

Sources: src/config.rs:415-430

Conclusion

Bounded Upstream Retry serves as a critical safety net in shunt, mitigating the impact of flaky upstream providers without compromising the streaming requirements of the Anthropic Messages protocol. By categorizing failures and enforcing a strict "before headers" boundary, it balances reliability with the low-latency expectations of LLM applications.

Sources: ARCHITECTURE.md:46-55, src/retry.rs:100-105