ci(cla-rerun): wait out a 403/429 spent budget, exit 1 when the reset is past the deadline

Merged
TE
24d ago
+331-37

The CLA quota re-run janitor ended green on a GitHub 403 while the CLA checks it exists to re-run stayed red. Coordinator ask: "It should treat a 403 (and a 429, or any response with X-RateLimit-Remaining 0) as a spent budget: wait until the reset in X-RateLimit-Reset, or Retry-After if that's given, then re-run the rejected CLA checks. Keep the wait bounded by the job's timeout. If the reset falls past the job's timeout, exit non-zero with the reset time, so the next scheduled tick picks it up instead of reporting green."

Evidence (2026-10-26T17:20Z to 2026-10-27T17:20Z, after #25525)

  • Janitor run 40421600026 (23:42Z) logged only ##[warning]HTTP Error 403: Forbidden and ended success, with no wait and no re-run.
  • Over that day, 40 quota-caused CLA episodes stayed red for more than 15 min. The worst was 37.9 min (#22901, guard red at 23:36Z, green at 00:14Z after the reset).

Change

  • GitHub.request turns an HTTP 403, an HTTP 429, or any error with X-RateLimit-Remaining: 0 into SpentBudget, which carries Retry-After and X-RateLimit-Reset. A successful response with X-RateLimit-Remaining: 0 stops the next call the same way. The free /rate_limit read is judged by its body: a workflow token at 0 that the pass cannot wait out is a spent budget, while one under the floor but above 0 still ends green, as before. Other HTTP errors stay warnings, as before.
  • main runs the pass again once the budget is back: after Retry-After when it is given, otherwise just after X-RateLimit-Reset (+20 s), otherwise after 60 s. This happens at most 3 times, all within PASS_DEADLINE (43 min; the step timeout is 44, the job timeout 47). Runs this job has already re-run are skipped when the pass runs again, and they take no slot. Every call checks the wall-clock deadline. A call that runs into the deadline exits 1 rather than ending as a warning, and a wait leaves 2 min of the pass free for the work after it.
  • When the resume time falls past the deadline, or the retries run out, the pass prints ::error with the time and exits 1. The job-level continue-on-error: true is removed so that the run shows red; the token-mint step keeps its own continue-on-error. The existing wait for a near reset is now also bounded by the deadline.

Before / after

  • Before: a 403 gives exit 0 and a green run, with no re-run.
  • After: a 403 with a reset 6 min out waits 380 s and then re-runs. A 429 with Retry-After: 30 waits 30 s. A reset past the deadline exits 1 with spent until HH:MM:SSZ.

Tests

  • tests/test_cla_quota_rerun.py has new tests: the 403 wait and re-run, the 429 Retry-After wait, remaining 0 on another status, the deadline exit 1, the retry cap, other HTTP errors staying warnings, and the workflow bound (step timeout above PASS_DEADLINE, no continue-on-error). With the fix, 36 of 36 pass. Without it, the new tests fail.

Review: Codex gpt-6-astra at xhigh, three rounds; SHIP at the head. Risk: low. The only workflow change is that a janitor run which cannot finish its pass now shows red. No new secrets, no permission change.

Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • CI Improvements
    • CI checks now retry eligible GitHub API rate-limit failures, using reset or retry-after information and respecting the run deadline.
    • A check fails when the required wait or retry cannot finish before the deadline, or when its retry limit is reached.
    • Other API errors continue to be handled separately from rate-limit failures, and reruns already completed during the job are not repeated.