ci(nightly-next): decide's light jobs take a free mini, Blacksmith last

TE
6d ago
+128-8

HOLD: superseded by #29864; do not land.

Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT"

What

resolve-nightly-cmux-tui-client and find-nightly-next-recovery need only decide, hold no environment and no secret, and run git, gh, curl, bash and stdlib Python. On every nightly-next run they took blacksmith-32vcpu-ubuntu-2404 with no fleet route (run 41254086065: both started on Blacksmith 32 vCPU at 23:53Z, right after decide finished on an owned mini).

decide (which on attempt 1 already runs on an owned mini) now ends with the same two steps build-nightly-app uses for its light jobs (#29436): mint the read-only routing token, then scripts/ci/portable_job_placement.py with NIGHTLY=1. Each of the two jobs takes a mini that is idle at that moment, or its Blacksmith Linux runner-set member right away. Both steps are continue-on-error and run after the decision, so a failed placement leaves the old route in place and never fails decide. A re-run of failed jobs keeps the Blacksmith route (light_attempt must equal the run attempt).

  • tests/test_ci_nightly_off_blacksmith.py: both jobs leave BLACKSMITH_LINUX_ONLY_DEBT, which goes from 32 to 30 jobs. The light-jobs-on-Linux test accepts only decide's gated placement term ahead of the Linux set and still refuses any macOS label.
  • tests/test_nightly_trusted_routing.py: new DecideLightJobsOnAFreeMini covers routing, a stale or missing placement, re-runs, forks, no environment or secret on the placed jobs, and the token's read-only permissions.
  • .github/runners.json: both jobs are portable-checks. scripts/ci/runner_routing_table.py accepts decide's placement output as the nightly mini term.

Audit: NEXT nightly runs-on, feat-cmux-next at 07efcedc

  • Owned Macs first, then Blacksmith macOS: decide (control runner or nightly slot, then Blacksmith Linux on the rescue re-run), build-nightly-app and build-nightly-ghostty-cli-helper (picker owned, then Blacksmith), build-sign-notarize-nightly, verify-*, signer-dmg-image (sign picker), sign-nightly-identity-legacy, verify-nightly-identity-legacy and generate-nightly-deltas (owned slot or side lane only, no Blacksmith).
  • A free mini first, then Blacksmith Linux: build-nightly-remote-daemon and await-nightly-tree-marker (#29436). This PR adds resolve-nightly-cmux-tui-client and find-nightly-next-recovery.
  • Blacksmith Linux only (30 jobs, the ratchet's debt): both promote jobs, the two pickers, build-nightly-inputs, resolve-nightly-ghostty-cli-helper, both prepare jobs, the signer-* Linux jobs, attest-nightly-remote-daemon, the visual-gate router, verdict and report, every publish job, release notes, the reel, and the failure reporting. Most of them hold an environment secret (publish, signing, routing) or produce Linux-only output (signer-build builds a Linux rcodesign), so each needs its own design and is not a mechanical move. #29821 just moved the DMG signer back to Linux for this reason.
  • Display Mac: nothing waits on one. visual-gate-nightly runs only when NIGHTLY_NEXT_VISUAL_GATE is set (#28961). The launch smoke is a separate post-publish workflow.

Prior art: #29436 and #29650 (same pattern, merged), #29821 (open, DMG signer, separate jobs). #28967's failures are already gone on base; the current test_ci_runs_on_fallback red is signer-build-macos, which #29821 removes.

Tests

Run locally: python3 -m unittest tests/test_ci_nightly_off_blacksmith.py tests/test_nightly_trusted_routing.py tests/test_ci_portable_job_placement.py tests/test_nightly_workflow_size.py tests/test_ci_runs_on_values.py tests/test_publish_app_token_workflow.py tests/test_ci_warm_sources.py tests/test_ci_linux_single_size.py tests/test_ci_signing_owned_mac.py, all OK. tests/test_runner_routing_table.py, tests/test_ci_runs_on_fallback.py and tests/test_ci_macos_xcode_selection.py fail the same way on base and on this head: signer-build-macos and signer-dmg, from #29801 and fixed by #29821. tests/test_runner_label_policy.py fails on base and here on CI_CONTROL_RUNNER (base red).

nightly.yml is now 501,561 bytes, under the 508,000 headroom line.

Risk: both jobs can now run on an owned mini (macOS). Their scripts already handle macOS (pin-cmux-tui.sh uses shasum there, for example), and test_ci_macos_xcode_selection already lists resolve-nightly-cmux-tui-client as a git, gh and Python control job.

Agent session: agent session
🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Routes resolve-nightly-cmux-tui-client and find-nightly-next-recovery to a free owned mini first, keeping Blacksmith as fallback, so nightly-next jobs no longer rely on Blacksmith by default.

  • decide mints a read-only routing token and runs portable_job_placement.py (NIGHTLY=1) as its last steps, the same pattern build-nightly-app uses for its light jobs.
  • Each placed job takes a mini idle when decide finishes, or its Blacksmith runner right away; failed or re-run jobs keep the old Blacksmith route.
  • Both placement steps are continue-on-error and run after decide's decision, so a failed placement never fails decide; decide's token has actions: read so the placement can read the queue janitor's snapshot, and both steps are registered in nightly-step-coverage.json.
  • Updates runners.json, runner_routing_table.py, and the placement/routing tests; Blacksmith-only debt drops from 32 to 30 jobs.
  • resolve-nightly-cmux-tui-client and find-nightly-next-recovery continue to hold no environment and no secret.