ci: fill idle and briefly busy owned minis before Blacksmith
Merged
TE
2mo ago+330+330+012345678901234567890123456789-85-85-01234567890123456789
On 2026-09-25 (08:30 to 22:50Z), Blacksmith ran about 35.6k macOS job-min. The PR picker sent 11.2k of those off the owned pool while owned runners sat idle: 12.7k idle unit-min overlapped Blacksmith work.
Change (scripts/ci/pr_runner_pool.py)
- Fleet first, Blacksmith as overflow. An owned pool now takes a run while its jobs start within
CI_PR_POOL_QUEUE_ROUNDSjob lengths. Blacksmith's expected wait no longer matters. Before, the allowance was capped at that wait, which comes from a snapshot up to 45 min old. It read 0 whenever a Blacksmith pool had looked free, so minis that were busy for a few minutes lost the run ("0 of 19 root runners free": 2.5k min). Blacksmith's wait still appears in the reason text. - Live runners decide, and in-flight peaks are only the fallback. With the route App token, a label is charged only for what holds it now: busy runners, the live window's runs, and, where no runner is idle, the janitor's queue plus one job for each older run.
committedand the markers' peaks for jobs that don't exist yet no longer fill the queue bound ("every pool full" at in-flight peaks: 2.6k min). Without the runners API, the old accounting still applies.Routednow counts runs per pool (owned_runs) for this. - A missing or stale snapshot no longer skips the fleet. On attempt 1 with live runners, a failed download (the R2 503) or a stale or missing snapshot now lets the live runners decide the owned pools. Blacksmith's queues are treated as unknown, and the reason and step summary say so. Before, the picker took the default route (2.8k + 0.6k min). Forks and retries are unchanged.
- The iOS docstring now matches. E2E and iOS share
pick(), so they get fleet-first too. The E2E docstring sentence will follow after #14584 lands, to avoid a conflict with it.
Replay on real runs from 2026-09-25
changes job logs and the janitor snapshots they read. Old = upstream/main, which reproduces the logged pick where the log's inputs are complete.
| run | needs (machines/root) | live idle std, root | old (upstream) | new |
|---|---|---|---|---|
| 36169149015 17:47Z | 10/9 full suite | 0/40, 0/18 | claude-wrapper owned; admission, 7 shards, lag, cli-product to Blacksmith | all 11 jobs owned |
| 36172350539 18:18Z | 10/9 full suite | 0/40, 0/18 | same as above | all 11 jobs owned |
| 36174151844 18:38Z | 3/2 | 0/40, 0/18 | swift-package owned; admission, shard-8, cli-product to Blacksmith | all owned |
| 36193075101 21:45Z | 3/2 | 17/42, 6/19 | swift-package owned; admission, shard-8, cli-product to Blacksmith | all owned |
| 36192259117 21:34Z | 10/8 | 16/42, 7/19 | admission, shard-1, 2 side lanes owned; 7 jobs to Blacksmith | all owned |
| 36181629965, 36182207707 | 3/2 | idle | all owned | all owned (unchanged) |
With the same inputs and a snapshot download that failed with 503, old takes the default route (Blacksmith 6vcpu) and new keeps every job on the owned pool.
Replay of the ci-dash "kept off idle roots" runs (2026-09-26, 01:09 to 01:22Z)
ci-dash listed 22 attempt-1 PR runs that the picker placed on Blacksmith while std runners were idle (20 to 27 of 42). Upstream reproduces 19 of the 22 logged picks. Across them:
- Blacksmith jobs drop from 111 to 35.
- 17 runs move entirely to owned minis, and 3 more move partly.
- Most were "least expected wait" picks after replaying a burst of about 26 newer runs, with the root queue at 16.
- A few of the flipped runs take the light tier, because std's root queue is full within its rounds.
Review follow-ups (second commit)
- Queue bound read live: it counts the peaks of the runs of the last 10 minutes (
DEFAULT_JOB_MINUTES) that took the pool, because their shards are on the way. Only runs of the last 3 minutes count toward the wait. Without this there is no queue signal when no snapshot exists. - Live-only overflow: with no snapshot, a run that no owned pool takes keeps every job's default route. Before this, it went to the first Blacksmith pool, whose queue is unknown.
- Warm keys: a stale snapshot's warm keys are kept for warm affinity.
- Docs:
docs/ci-runners.mdno longer says the owned rule waits on Blacksmith's.
Tests
tests/test_ci_pr_runner_pool.py: 197 pass. Four tests are updated to fleet-first. The newLiveIdleRunnersclass covers: committed peaks ignored live but kept as the fallback, a 503 or a stale or missing snapshot, retries and forks unchanged, the step summary, and run counting.- Also passing:
test_run_e2e,test_ci_owned_pool_rescue,test_ci_queue_janitor,test_ci_fork_runner_routing,test_runner_label_policy,test_ci_owned_warm_state, andscripts/ci/guards-local.sh.
The kill switch is CI_PR_POOL_QUEUE_ROUNDS=0 (the old rule). Part of manaflow-ai/cmuxterm-hq#661.
🤖 Generated with Claude Code
Summary by CodeRabbit
- Improvements
- CI runs can use owned runner pools when they fit the configured queue allowance, independently of the expected wait on hosted runners. The queue limit remains based on pool size and configured rounds.
- Runner selection can use live capacity when available. Missing, outdated, or unreadable capacity snapshots no longer prevent a run from being considered for an owned pool.
- Recent run activity is included in queue estimates, and default routing is preserved when no owned pool is selected.
- Documentation
- Updated guidance on owned-pool selection and queue limits.