perf(canary-rollout): date-bound the Agent Ingress run list (the dominant gate cost)

Merged
DO
1mo ago
+223-6

Summary

Second half of the plan in issue 1259 (Refs below), driven by the first scheduled run that carried the CANARY_TIMING output from #1260 (run 37581060882):

gate stepwalltime inside ghscript timegh run list callstheir totalavg / max
promote-all745 s740 s (99%)6 s58607 s10.5 s / 31 s
sync-issues769 s763 s (99%)6 s58593 s10.2 s / 30.5 s

Every other gh call kind is cheap (tag lookups 0.38 s avg, matching-refs 0.43 s, ...). gh run list is ~80% of each gate step.

Cause (from the code; the timing kind could not yet say which workflow). The Agent Ingress run list of each collapsed repo is read with -L 5000 (CANARY_INGRESS_RUN_LIMIT), so gh pages through up to ~50 pages per repo, although every window the gate samples starts at most ~2 weeks back (the 14-day baseline is the widest by default). That list was introduced in #1238.

Fix. The ingress list is now fetched with --created ">=<date>":

  • _ingress_horizon <since> snaps the window start to a small set of day tiers (CANARY_INGRESS_HORIZON_TIERS, default 8 15: the candidate/correctness windows and the 14-day baseline). Older or empty windows, and a SOAK_WINDOW_DAYS beyond the largest tier, stay unbounded, exactly as before.
  • _repo_wf_runs_cached takes an optional 5th arg (the date). It is part of the cache key, so bounded and unbounded lists never serve one another, and windows in the same tier share one fetch.
  • GitHub caps any workflow-run listing that uses created at 1,000 results (found in review). _ingress_runs_fetch therefore trusts a bounded list that reaches 1,000 runs only if its oldest run is before the window start (everything newer is then in it); otherwise it reads the unbounded list, i.e. exactly the previous behavior. For such very busy repos there is no saving, and no loss of history.
  • The timing report now labels run lists as run list [<workflow>] L=<cap> [bounded], so the next scheduled run shows directly whether the ingress lists were the slow ones and whether the bound removed the cost.

What this does not change (semantics)

  • Which runs count: the list is only trimmed of runs older than every window that uses it; each caller still cuts at its own since. The horizon date is the start of the day t days back, with t >= the window age in days rounded up, so it is never later than the window start (tested for 0-15 days back).
  • The fail-closed cap guard is untouched: a bounded list that hits CANARY_INGRESS_RUN_LIMIT before the window start is judged exactly like an unbounded one (tested), and the 1,000-result ceiling above is handled by the fallback.
  • Per-role (legacy) workflow lookups are not bounded.

What this does not prove

I could not measure ingress run volumes offline (the repos' Actions data is not readable from this session), so the size of the saving is an estimate: if the lists are dominated by runs older than 14 days, the 10 s calls should fall to ~1 s, i.e. up to ~9 min per gate step. If the volume is mostly inside 14 days, or a repo exceeds 1,000 runs in its window, the saving will be smaller for it. The relabelled timing output from the next scheduled run settles it, and issue 1259 stays open until the post-fix medians are known.

Risk

Low-medium: it changes what the promotion gate fetches. Mitigations: the bound is conservative (rounded up to a tier, unbounded beyond), a possibly-truncated bounded list is never used, the existing ingress tests are unchanged and green, and new ones pin the new behavior. Rollback is a revert.

Test plan

  • New bats tests: tier mapping (including the empty-tiers disable) and the "never later than the window start" invariant; bounded for a recent window, unbounded for an old/empty one; per-tier cache sharing (3 fetches for 4 windows, none repeated); cap guard on a bounded list; the 1,000-result fallback and the "reaches back, trusted" case; run-list timing labels. One earlier #1260 timing test updated for the new, more specific run list label.
  • Full suite 448/448, and again with CANARY_PRELOAD_REGISTRY=1. shellcheck --severity=warning clean.

Rollback

Revert this PR, or set CANARY_INGRESS_HORIZON_TIERS="" (empty) on the workflow step to disable the bound without a revert.

Monitoring

On the next scheduled Canary Rollout runs read the canary timing block: run list [Agent Ingress] L=5000 bounded calls/avg/max, and the two gate steps' wall time (10.4 min before #1238, 15.6 after, 12.4/12.8 in the first sample with #1260).

Refs #1259, #1260, #1238.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MjXPsBTaL6WV9AhCH2ajuv