feat(evals): per-provider concurrency under the global cap

Merged
MI
8d ago
+1555-24

Stack

1 of 4 in the evals TUI overhaul. Base: main.
→ #PR1 · #PR2 config v2 + env · #PR3 doctor · #PR4 run experience

Summary

Rows now wait for a per-provider slot before they run, under the global -c cap. One provider can no longer take every slot and then 429 on all of them at once.

  • ProviderConcurrency (framework/providerConcurrency.ts): one semaphore per model provider, 3 wide by default, clamped to -c. Set widths with EVAL_PROVIDER_CONCURRENCY=openai=6,anthropic=4. The cap applies within one process; separately launched runs don't share it. Cross-cell capping belongs to the matrix planner (follow-up).
  • Backpressure: when the agent's own call hits a 429 or a connect timeout, that provider's width is halved once per 60 s window and the row is retried once (runGatedRow). A Browserbase session-create 429 is retried once after 20 s. A rate-limited judge never throttles and never re-runs the agent. Retried rows are tagged provider_throttled.
  • started fires when the slot is acquired, so a queued row never shows as running. planned and the progress total now count testcases × trials.
  • Queue events: the runner emits queue and throttled events (the board uses them in #PR4), and the abort path marks queued rows finished.
  • Header: shows the widths that will actually gate the run.
  • Browserbase: session-creation errors now include the API's detail instead of a bare failure.
  Env: BROWSERBASE  Harness: codex  Tool: stagehand_facade  Concurrency: 10 global · openai 3

[!WARNING]
Behaviour change: a single-provider cell at -c 10 now runs 3-wide unless you raise the provider width (EVAL_PROVIDER_CONCURRENCY=openai=10).

Test plan

  • vitest packages/evals/tests: 841 passing. New: providerConcurrency.test.ts (semaphores, halving window, retry rules, judge exemption, abort while queued) and runnerProgressEvents.test.ts (a row queued behind openai=1 is not reported as started).
  • tsc, fmt:check

Summary by cubic

Adds per-provider concurrency gating so rows no longer let one provider consume the entire global -c cap and then 429 en masse.

  • Rows wait for a per-provider slot before running; each provider defaults to 3 concurrent executions, tunable via EVAL_PROVIDER_CONCURRENCY=openai=6,anthropic=4. A malformed value fails the plan, --dry-run/--preview included.
  • An agent-side 429 or connect timeout halves that provider's width once per 60 s window and retries the row once; a Browserbase session-create 429 retries once after 20 s (the wait ends on abort); a rate-limited judge never re-runs the agent. Retried rows are tagged provider_throttled.
  • Testcases are interleaved by provider before dispatch, rounded in proportion to each provider's width, so one provider's queue can't strand global slots while other providers idle.
  • Browserbase create errors now surface only the HTTP status, and a provider-less row no longer treats a 429 in its own output as a throttle.
  • started now fires only once the provider slot is acquired; the runner also emits queue and throttled events, and the run header shows the effective provider widths. Options include a new providerConcurrency config source that env widths layer over.

Behavior change

A single-provider cell at -c 10 now runs 3-wide by default; raise it with EVAL_PROVIDER_CONCURRENCY=openai=10 to restore the old width.