fix(providers): keep failed model discovery from opening the circuit breaker

Merged
SA
1d ago
+187-12

Failed /models calls counted toward the same circuit breaker as chat. While a provider's listing was down, chat through the gateway got 503 circuit breaker is open even though the provider's chat worked. Listing retries once a minute, so about five minutes of failures with no successful call in between was enough to open it. Fireworks hit this during release testing, when its /models returned 500 and chat stayed healthy.

Model discovery and availability probes now bypass the breaker through llmclient.WithoutCircuitBreaker, set once in the registry so it covers every provider.

User-visible impact

  • A listing outage no longer pauses inference. Provider-qualified requests keep working, and the outage still shows on the provider status page.
  • An open breaker no longer blocks listings, and a successful listing no longer closes a breaker that inference failures opened. Only inference traffic moves the breaker now.
  • If inference failures opened the breaker on a provider that then goes idle, the status page shows "Circuit Open" until the next inference request, because listings no longer update that state. Routing does not read this value, and the breaker still recovers on the next request after the timeout.

Models from a provider whose listing is failing stay hidden from /v1/models, as before.

Testing

  • TestFailedDiscoveryDoesNotOpenInferenceCircuitBreaker runs a refresh and an availability probe through a real llmclient. It fails without the registry change.
  • Against a fake upstream with /models returning 500, chat stayed at 200 and reached the upstream. On v0.2.0 it returned 503 after about six failed listings.
  • Against the real Fireworks API during its listing outage, chat and streaming both returned 200 after seven failed listings.

Summary by CodeRabbit

  • Bug Fixes

    • Model discovery and availability checks no longer affect the inference circuit breaker: failed model listings won’t open it, and successful listings won’t close it.
    • Provider status continues to report failed model listings. With model-scoped breakers, model discovery bypasses the breaker; requests without a model continue to use a separate provider breaker.

  • Documentation

    • Updated resilience guidance in English, German, Polish, and Chinese.