test: run the bats suites on macOS with scripts/test-local.sh
Why
Agents and people keep reporting that tests "can't run locally on the Mac". I measured it on main (8917aa72b), with all 263 bats files each run on their own:
| Setup | Failing tests | Failing files |
|---|---|---|
| Stock macOS (bash 3.2, BSD sed/grep/date/awk) | 2,039 | 124 |
| Homebrew bash 5 + coreutils + gawk first on PATH | 160 | 25 |
This PR: scripts/test-local.sh | 0 of 6,361 | 0 |
Every residual failure was environmental: no GNU sed (sed -i, sed -z), no GNU grep (grep -P), no flock, no jsonschema in Homebrew python, and /var → /private/var in test_pr_worktree.bats.
What changes
Brewfilelists the GNU toolchain. Onebrew bundle --file=Brewfileinstalls it.scripts/test-local.shworks on macOS and Linux:- On macOS it puts the GNU tools and bash 5 first on PATH for that run only, and resolves
TMPDIRto its real path. - When
python3can't import jsonschema and yaml, it builds a cached venv (outside the checkout, so worktrees share it) from the hash-pinnedrequirements/ci/files. - It then runs bats. With no paths it runs every suite, which covers more than CI's lists.
--jobs Nruns files in parallel; other options go straight to bats. - The script itself starts under bash 3.2.
- On macOS it puts the GNU tools and bash 5 first on PATH for that run only, and resolves
tests/dev-lead/unit/toolchain_preflight.bash, called fromsetup_suite, checks the toolchain on macOS only.- A bare
batsrun without the toolchain now stops with one message naming what is missing and pointing toscripts/test-local.sh, instead of failing thousands of tests. - Linux and CI are not checked.
BATS_SKIP_TOOLCHAIN_PREFLIGHT=1skips it. test_toolchain_preflight.batscovers it. A counterfactual where the check never fails makes tests 2 and 3 fail.
- A bare
scripts/lib/comment-disposition-verify.sh:_cdv_has_tokenescaped/in an ERE. GNU grep 3.12 printsgrep: warning: stray \ before /to stderr for that, which failed threetest_comment_disposition_verify.batstests. CI's grep 3.11 is quiet today./isn't special in an ERE, so the match is unchanged.- AGENTS.md (Local environment caveats) and CLAUDE.md (Commands) now document the one command in place of "cannot run locally".
Verification
- macOS 26.5, Apple Silicon:
scripts/test-local.sh --jobs 4ran the full suite: 6,352 of 6,355 passed in 32 minutes. The three failures were the grep warning.- After the fix, those suites plus
test_fix_reviews.bats(340 tests) and the new preflight test: 471 of 471 passed.
- Stock
bats tests/test_ci_status.batsnow stops insetup_suitewith a message naming bash 3.2, BSD sed, BSD grep, BSD date and BSD awk. /bin/bash scripts/test-local.sh …(bash 3.2) works. With a fakeHOMEBREW_PREFIXit exits 1 and names the missing formulae.- shellcheck (warning),
check-bats-negation.sh,check-fixed-tmp-paths.shand markdownlint all pass. - Not verified on Linux locally; CI on this PR covers that.
🤖 Generated with Claude Code
Problem
No linked issue: see the Why section above (2,039 failing tests on stock macOS, 160 with the existing PATH workaround, 0 with this PR).
Risk
Low — changes automation shell logic under scripts/, covered by shellcheck (--severity=warning) and the bats suite.
Test plan
Tests added/updated: tests/dev-lead/unit/setup_suite.bash,tests/dev-lead/unit/test_toolchain_preflight.bats tests/dev-lead/unit/toolchain_preflight.bash. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.
Rollback
Revert this PR. No non-revertible side effects (no tags, migrations, or external state).
Monitoring
This PR's Lint (shellcheck) and bats checks show pass/fail; watch subsequent dev-lead / pr-review runs for behavioral regressions.