Account Pooling & Quotas

Relevant source files

The following files were used as context for generating this wiki page:

• src/accounts.rs
• src/usage.rs
• src/usage_poll.rs
• src/state_persist.rs
• ARCHITECTURE.md
• README.md

Account Pooling and Quotas in shunt provide a reactive and proactive system for managing multiple upstream LLM credentials. The system allows the gateway to rotate through a pool of accounts (such as Anthropic or Codex) based on observed usage, sticky session requirements, and remaining rate limits. By tracking 5-hour and 7-day usage windows, shunt ensures that inference requests are directed to healthy, available accounts while avoiding those nearing their quota limits.

The architecture emphasizes "reactive load balancing" and "proactive rotation." Reactive measures include handling 429 (Too Many Requests) or 401 (Unauthorized) errors by placing accounts in temporary cooldowns. Proactive measures involve monitoring utilization headers and polling upstream usage APIs to prevent account exhaustion before it occurs.

Sources: ARCHITECTURE.md, README.md

Core Architecture and Data Flow

The pooling system is managed via the accounts module, which maintains the state of all configured provider accounts. It interacts with the usage module to process quota information extracted from upstream response headers or polled from management APIs.

Account State Management

The accounts.rs module manages a collection of accounts, each identified by an AccountKey (combining provider and identity). It tracks utilization percentages for various time windows and handles account "stickiness" using session IDs.

The diagram above shows the lifecycle of an account selection within a single request. Sources: src/accounts.rs, ARCHITECTURE.md, src/state_persist.rs

Key Data Structures

Structure	Description	Source
AccountKey	Unique identifier for an account, containing provider and identity.	src/accounts.rs
QuotaState	Holds utilization percentages and reset timestamps for 5h and 7d windows.	src/accounts.rs, src/state_persist.rs
AccountSnapshot	A point-in-time view of an account's health, including utilization and cooldown status.	src/accounts.rs
PoolConfig	Configuration for thresholds, polling intervals, and persistence paths.	README.md

Quota Tracking and Proactive Rotation

shunt tracks three primary utilization windows: 5-hour, 7-day, and 7-day Over-Inference (7d_oi). Utilization is expressed as a decimal (0.0 to 1.0).

Selection Logic

The select_order function generates a prioritized list of accounts for a request. Accounts are filtered and sorted based on:

1. Availability: Accounts in a reactive cooldown (due to 429/5xx errors) are deprioritized or skipped.
2. Sticky Sessions: If an x-claude-code-session-id is present, the system attempts to use the last successful account for that session unless it has crossed a "near-quota" threshold.
3. Burn-Rate Ordering: Accounts are sorted by their remaining headroom to balance usage across the pool.

Sources: src/accounts.rs, README.md

Usage Polling

For accounts that support it (specifically refreshable Anthropic OAuth accounts), shunt can poll the upstream usage API. This reconciles out-of-band consumption that occurs outside of the gateway.

The poller ensures that the gateway has an accurate view of account health even if requests are made directly to the provider. Sources: src/usage_poll.rs, src/usage.rs

Persistence and Warm-Starts

To prevent a "cold start" where all accounts appear to have 0% utilization after a gateway restart, shunt supports opt-in on-disk persistence of quota states.

Persistence Mechanism

When [server.pool] state_path is configured:

• Restore: At boot, state_persist::restore reads the JSON state file and populates the account pool.
• Flush: A background task periodically checks if the pool is "dirty" (has updated quota information) and writes a versioned JSON envelope to disk.
• Atomic Writes: To prevent corruption, the system uses atomic writes (writing to a temporary sibling file and renaming it).

Sources: src/state_persist.rs

Persistence Configuration

Option	Type	Description	Default
state_path	Path	File path to store the JSON quota state.	None (Disabled)
FLUSH_INTERVAL	Duration	Internal constant; writes dirty state every 15 seconds.	15s

Sources: src/state_persist.rs

Client Usage Endpoints

shunt exposes sanitized endpoints for monitoring pool health without requiring administrative access.

GET /usage

Returns a coarse status of the pool (ok, degraded, or exhausted) and remaining headroom. It authenticates using the same client token as the inference routes.

• Privacy: It never exposes specific account names, counts, or internal priority thresholds.
• Status Mapping:
  • ok: Sufficient headroom across the pool.
  • degraded: Some accounts are near quota or cooling.
  • exhausted: No accounts available for the request.

Sources: src/usage.rs, README.md

GET /api/oauth/usage

An emulation of the Anthropic OAuth usage API. This allows the Claude Code CLI to render native usage bars (current session/week) based on the combined health of the shunt pool.
Sources: README.md, src/usage.rs

Summary

The Account Pooling and Quota system transforms shunt from a simple proxy into a robust management layer for LLM resources. By combining reactive error handling with proactive utilization tracking, persistence, and automated polling, it provides high availability for inference tasks while strictly respecting upstream rate limits. Use of [server.pool] configuration allows operators to tune thresholds and enable persistence for stable, production-ready gateway operations.