ci(runners): the linux pool runs smallest first (4, 8, 16 vCPU), so LINUX_RUNNER stops putting every small job on 32 vCPU

Merged
TE
5d ago
+70-44

The linux runner pool (.github/runners.json pools.linux) now lists blacksmith-4vcpu-ubuntu-2404, then blacksmith-16vcpu-ubuntu-2404, smallest first. It used to be 32 then 16 vCPU. That makes RUNNER_POOLS.linux 4 vCPU, and so is its LINUX_RUNNER mirror (runner_pools.py). Every job on main and feat-cmux-next that reads either one now gets 4 vCPU.

Leo: "ok. we clearly are not able to use them. so we should have resized a long-ass time ago." and "(im pretty sure blacksmith trimmed us off a bit just cuz we were allegedly costing like $10k a day) so we need to figure out solulus"

Why

Blacksmith now holds us to a small, hard concurrency cap counted in vCPUs, and the 32 vCPU label had about 1,531 jobs queued. Because the pool's first member was 32 vCPU, hundreds of jobs that take a tenth of a minute (pr-perf-section-check, pr-capture-check, ci-status, guard status) each took a 32 vCPU slot. That is the room of eight 4 vCPU jobs. Proof that LINUX_RUNNER was 32 vCPU: pr-perf-section-check (vars.LINUX_RUNNER || 'blacksmith-2vcpu-ubuntu-2404') ran 2,940 times on 32 vCPU on 2026-10-31. Every member is still probed, and when 4 vCPU stalls the pool moves to 16 vCPU (runner_pools.py, unchanged here). 32 vCPU is out of the pool: under a vCPU cap it never starts when 4 and 16 cannot. 8 vCPU is out too. 47 bounded selectors (contains([4,16,32], vars.LINUX_RUNNER) && vars.LINUX_RUNNER || 4) refuse 8 vCPU, so if the pool stepped onto 8, their jobs would stay on the stalled 4 vCPU label. That came out in review. A new test, test_every_bounded_selector_accepts_every_linux_member, now fails on any member a bounded selector refuses: 47 subtests fail with 8 vCPU in the pool.

Changes

  • .github/runners.json: pools.linux.members is now 4 then 16 vCPU, and its why gives the measurement.
  • scripts/ci/cloud_overflow_switch.py PROBE_LABELS and the probe matrix in ci-cloud-overflow-probe.yml are the same two sizes. tests/test_ci_runner_pools.py already requires the probe set to equal the pool's members. ALTERNATE_FAILOVERS[LINUX_FAILOVER] is now 4 vCPU (it was 32): toward_started() only picks a probed size, and the bounded selectors accept 4. runner_pools.py is untouched.
  • Tests that pinned the old order or the old probe set are updated: test_ci_runner_pools, test_ci_cloud_overflow_switch, test_ci_fork_runner_routing. The 47 bounded selectors still list 32 vCPU, so the fork-routing test now accepts a label declared in runners.json or .github/actionlint.yaml. docs/ci-runners.md names the new probe set.

Performance

Source: the HQ build-fleet per-job data for 2026-10-31 (one row per job: workflow, job, label, start and end). Prices are from HQ build-fleet/swift-linux-builder/blacksmith/RESULTS.md (blacksmith runners catalog): 4 vCPU $0.008/min, 32 vCPU $0.064/min, linear. The "after" column is a projection, not a measurement. I evaluated every job's runs-on on main and feat-cmux-next with LINUX_RUNNER / RUNNER_POOLS.linux set to 32 and then to 4. The jobs that move are billed at 4 vCPU. Where a job also ran on 4 vCPU on 2026-10-30 (when LINUX_RUNNER was 4 vCPU), its 4 vCPU wall time from that day is used; otherwise its wall time is kept. Most of these jobs are bound by setup time and do not get slower.

metricbefore (2026-10-31)after (projected)deltasource
jobs/day on blacksmith-32vcpu-ubuntu-2404106,443about 42,700-63,699HQ build-fleet job data, 10-31
billed min/day, 32 vCPU193,021about 114,200-78,845same
billed min/day, 4 vCPU27,952about 112,500+84,506same, 4 vCPU wall time from 10-30
$/day, 32 vCPU$12,353about $7,307-$5,046RESULTS.md prices
$/day, 4 vCPU$224about $900+$676same
$/day, all Blacksmith$19,303about $14,933-$4,370 (-23%)same
Linux vCPU-minutes/day (what a vCPU cap counts)8.82M6.64M-25%same

Jobs that read the pool or LINUX_RUNNER and take longer than a minute

  • cmux-next guards / release-notary: 5.7 min on 32 vCPU, 6.2 on 4 (10-30). Bound by setup time.
  • cmux-tui artifacts / publish to R2: 1.5 to 4.7 min. release rails: 3.2 to 4.9 min. cmux-tui SDKs / seven-language live conformance: 4.1 min on 32 vCPU, about 7 on 4 (n=7), with a 45-minute timeout. All three stay well inside their timeouts.
  • fast-checks.yml checks (timeout 195): takes inputs.runner (8, 16 or 32 vCPU, from the broker's reroute) when one is given. Without one it reads the pool and gets 4 vCPU. Mean 3.1 min on 32 vCPU.
  • main's ci-web.yml react-apps-check (timeout 10) reads LINUX_RUNNER. PRs into main are rare. If it times out on 4 vCPU, it should name a size (follow-up). On feat-cmux-next the heavy react-apps-check (N) shards are react-apps-check-shard, a literal 32 vCPU job that this PR does not change.
  • r2-binaries-prune already reaches its 60-minute timeout on 32 vCPU, so the runner size is not what makes it slow.

Left as follow-up: literal 32 vCPU, not affected here (about 110k min and $7.05k per day still on 32 vCPU)

All of these write !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || ... || 'blacksmith-32vcpu-ubuntu-2404' or similar on feat-cmux-next:

  • the react-apps-check shards (about 33k min/day)
  • cmux-tui-build-package builds (14.5k)
  • cmux-next CI guard group (8.1k)
  • cmux-next fast tier (cargo) (7.0k)
  • cmux-tui tree preflight (3.0k)
  • react-apps-base-check (2.6k)
  • tooling-tests (2.0k)

Each of these should get a size picked from its own measurements, in its own PR.

After merge

Until the next reconcile (every 10 min), ci-repo-variables' runner_pools.py --check may fail once on main, because the live variables still name 32 vCPU, which is no longer a member. It clears itself on that reconcile.

Overlap

  • #30308 (runner_pools mirror_writes: LINUX_RUNNER leaves a dead size) pairs with this PR. With #30308, LINUX_RUNNER follows the pool again. With this PR, the pool's live member is small. Both change tests/test_ci_runner_pools.py: #30308's new test reads the real runners.json with 32 vCPU as a member, so whichever PR lands second adapts that test.
  • #29844 and #30277 move small jobs from LINUX_RUNNER onto the pool. With this PR those jobs land on 4 vCPU instead of 32.
  • Branch lane/pool-size-stall-failover (pushed, no PR yet) orders the pool largest first, 32/16/8/4. That design conflicts with this one, and I have told its coordinator. Smallest first is the point of this PR: the cap counts vCPUs, so the live member has to be small.
  • pr-overlap also lists #30076 and #27114, which touch the same files for other reasons.

Agent session: agent session

šŸ¤– Generated with Claude Code

Summary by CodeRabbit

  • Updates
    • Updated the Linux CI runner pool to prioritize 4-vCPU runners, with 16-vCPU runners as an alternative.
    • Expanded overflow checks to cover all Linux runner sizes in the pool, improving visibility into runner availability.
    • CI routing remains unchanged when an overflow check stalls, while health results are still recorded for individual runner sizes.