fix(providers): retry transient failures on OpenAI Responses and Gemini

Merged
WA
1mo ago
+870-79

Summary

  • OpenAI Responses (and Azure Responses endpoints) sent each request once, so a single transient 500 failed the block. Every OpenAI-compatible, Anthropic, and Bedrock client already retried through its SDK — this path and Gemini/Vertex were the gaps
  • New providers/retry.ts: one retry policy for calls no vendor SDK retries, matching what openai, @anthropic-ai/sdk, and the AI SDK agree on
    • PROVIDER_MAX_RETRIES (2) replays with jittered exponential backoff
    • retries 408/409/429/5xx and dropped connections (Node TypeError, Bun ECONNRESET)
    • obeys x-should-retry, then retry-after-ms, then retry-after
    • never retries a caller abort; covers only the wait for headers, so a started stream is never replayed
    • retries only real network failures, recognised by the socket error code anywhere in the cause chain (Node's fetch failed carries it on cause; Bun sets ECONNRESET / ConnectionRefused / ConnectionClosed directly), never by message text. A TypeError from building the request is a bug and is not retried
    • two deliberate divergences from the SDKs, both so a block's fallback model runs instead of waiting on a failure that will not clear: a 429 reporting an exhausted balance (insufficient_quota) is not retried, and neither is a failure whose requested delay exceeds 30s (a daily quota, a provider-wide pause) — the AI SDK likewise ignores long server delays
    • 429 bodies are read directly (bounded, under the caller's signal), not via clone(): cancelling one branch of a tee waits on the other, so an oversized body could hang
    • retry log lines carry status and x-request-id, and bound SDK error text to 200 chars
  • OpenAI Responses uses fetchWithProviderRetry; the payload is prepared once outside the retry, since preparing can compact the conversation with its own model call
  • Gemini/Vertex wrap generateContent/generateContentStream in withProviderRetry, honoring the google.rpc.RetryInfo delay from the error body (the SDK exposes no headers). The SDK's own retryOptions stays off: it replaces every error (4xx included) with a generic statusText message and ignores the abort signal. Deep research interactions.* is left alone — that client already retries, and wrapping would double it
  • Anthropic, Azure Anthropic, and Azure OpenAI chat pin maxRetries to PROVIDER_MAX_RETRIES (Azure OpenAI via openAICompatTransport()), so an SDK bump cannot silently move their retry budget. No behavior change: both equal the SDK defaults. Anthropic pins only maxRetries; its SDK derives the timeout from max_tokens
  • TypeSafe's existing loop now reads the shared status and retry-after helpers
  • Quota-body classifier moved from embeddings to lib/core/errors/provider-quota.ts (now two consumers)
  • Generation-context coverage guard accepts a const bound to a prepared payload, and follows the postOnce → post rename (the old name silently dropped the OpenAI body from coverage)

Accepted risk

  • A transport failure after the server finished a long non-streaming generation re-runs it, the same trade-off every SDK-backed provider already makes. /v1/responses takes no idempotency key. Bun's ~300s socket wall surfaces as TimeoutError, which is not retried, so a runaway generation is never replayed

Follow-ups (separate PRs)

  • Wand and vision call OpenAI with raw fetch and no retry; wrap them in fetchWithProviderRetry
  • OpenAI Responses and Gemini have no per-attempt time-to-headers bound (pre-existing)

Type of Change

  • Bug fix

Testing

  • providers/retry.test.ts: one test per failure mode (transient status, budget exhaustion, permanent 4xx, exhausted quota, x-should-retry both ways, retry-after-ms, dropped connection, unknown error, abort during backoff, pre-aborted request, SDK status errors); each guard mutated and its test watched go red
  • OpenAI and Gemini wiring tests, both red with the retry unwired; coverage guard verified to still flag an unprepared Gemini payload and OpenAI body
  • Every guard in providers/retry.ts mutated and its test watched go red (long delay on both paths, TypeError narrowing, Bun codes, uncapped retry-after, 429 tee hang, abort during body read)
  • Full providers + lib/embeddings suites (1,958 tests)
  • Independent audit of every provider's transport (18 OpenAI-compatible, Anthropic, Azure, Bedrock, Gemini/Vertex, deep research, TypeSafe): no double retries, abort signal passed everywhere, no replay after a stream starts
  • bun run type-check, bun run lint, bun run check:audits (51 audits), docs-manifest:check

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)