ci(cla-rerun): wait out a 403/429 spent budget, exit 1 when the reset is past the deadline
The CLA quota re-run janitor ended green on a GitHub 403 while the CLA checks it exists to re-run stayed red. Coordinator ask: "It should treat a 403 (and a 429, or any response with X-RateLimit-Remaining 0) as a spent budget: wait until the reset in X-RateLimit-Reset, or Retry-After if that's given, then re-run the rejected CLA checks. Keep the wait bounded by the job's timeout. If the reset falls past the job's timeout, exit non-zero with the reset time, so the next scheduled tick picks it up instead of reporting green."
Evidence (2026-10-26T17:20Z to 2026-10-27T17:20Z, after #25525)
- Janitor run 40421600026 (23:42Z) logged only
##[warning]HTTP Error 403: Forbiddenand endedsuccess, with no wait and no re-run. - Over that day, 40 quota-caused CLA episodes stayed red for more than 15 min. The worst was 37.9 min (#22901, guard red at 23:36Z, green at 00:14Z after the reset).
Change
GitHub.requestturns an HTTP 403, an HTTP 429, or any error withX-RateLimit-Remaining: 0intoSpentBudget, which carriesRetry-AfterandX-RateLimit-Reset. A successful response withX-RateLimit-Remaining: 0stops the next call the same way. The free/rate_limitread is judged by its body: a workflow token at 0 that the pass cannot wait out is a spent budget, while one under the floor but above 0 still ends green, as before. Other HTTP errors stay warnings, as before.mainruns the pass again once the budget is back: afterRetry-Afterwhen it is given, otherwise just afterX-RateLimit-Reset(+20 s), otherwise after 60 s. This happens at most 3 times, all withinPASS_DEADLINE(43 min; the step timeout is 44, the job timeout 47). Runs this job has already re-run are skipped when the pass runs again, and they take no slot. Every call checks the wall-clock deadline. A call that runs into the deadline exits 1 rather than ending as a warning, and a wait leaves 2 min of the pass free for the work after it.- When the resume time falls past the deadline, or the retries run out, the pass prints
::errorwith the time and exits 1. The job-levelcontinue-on-error: trueis removed so that the run shows red; the token-mint step keeps its owncontinue-on-error. The existing wait for a near reset is now also bounded by the deadline.
Before / after
- Before: a 403 gives exit 0 and a green run, with no re-run.
- After: a 403 with a reset 6 min out waits 380 s and then re-runs. A 429 with
Retry-After: 30waits 30 s. A reset past the deadline exits 1 withspent until HH:MM:SSZ.
Tests
tests/test_cla_quota_rerun.pyhas new tests: the 403 wait and re-run, the 429Retry-Afterwait, remaining 0 on another status, the deadline exit 1, the retry cap, other HTTP errors staying warnings, and the workflow bound (step timeout abovePASS_DEADLINE, nocontinue-on-error). With the fix, 36 of 36 pass. Without it, the new tests fail.
Review: Codex gpt-6-astra at xhigh, three rounds; SHIP at the head. Risk: low. The only workflow change is that a janitor run which cannot finish its pass now shows red. No new secrets, no permission change.
Agent session: agent session
🤖 Generated with Claude Code
Summary by CodeRabbit
- CI Improvements
- CI checks now retry eligible GitHub API rate-limit failures, using reset or retry-after information and respecting the run deadline.
- A check fails when the required wait or retry cannot finish before the deadline, or when its retry limit is reached.
- Other API errors continue to be handled separately from rate-limit failures, and reruns already completed during the job are not repeated.