test: run the bats suites on macOS with scripts/test-local.sh

Merged
DO
5d ago
+328-7

Why

Agents and people keep reporting that tests "can't run locally on the Mac". I measured it on main (8917aa72b), with all 263 bats files each run on their own:

SetupFailing testsFailing files
Stock macOS (bash 3.2, BSD sed/grep/date/awk)2,039124
Homebrew bash 5 + coreutils + gawk first on PATH16025
This PR: scripts/test-local.sh0 of 6,3610

Every residual failure was environmental: no GNU sed (sed -i, sed -z), no GNU grep (grep -P), no flock, no jsonschema in Homebrew python, and /var → /private/var in test_pr_worktree.bats.

What changes

  • Brewfile lists the GNU toolchain. One brew bundle --file=Brewfile installs it.
  • scripts/test-local.sh works on macOS and Linux:
    • On macOS it puts the GNU tools and bash 5 first on PATH for that run only, and resolves TMPDIR to its real path.
    • When python3 can't import jsonschema and yaml, it builds a cached venv (outside the checkout, so worktrees share it) from the hash-pinned requirements/ci/ files.
    • It then runs bats. With no paths it runs every suite, which covers more than CI's lists. --jobs N runs files in parallel; other options go straight to bats.
    • The script itself starts under bash 3.2.
  • tests/dev-lead/unit/toolchain_preflight.bash, called from setup_suite, checks the toolchain on macOS only.
    • A bare bats run without the toolchain now stops with one message naming what is missing and pointing to scripts/test-local.sh, instead of failing thousands of tests.
    • Linux and CI are not checked. BATS_SKIP_TOOLCHAIN_PREFLIGHT=1 skips it.
    • test_toolchain_preflight.bats covers it. A counterfactual where the check never fails makes tests 2 and 3 fail.
  • scripts/lib/comment-disposition-verify.sh: _cdv_has_token escaped / in an ERE. GNU grep 3.12 prints grep: warning: stray \ before / to stderr for that, which failed three test_comment_disposition_verify.bats tests. CI's grep 3.11 is quiet today. / isn't special in an ERE, so the match is unchanged.
  • AGENTS.md (Local environment caveats) and CLAUDE.md (Commands) now document the one command in place of "cannot run locally".

Verification

  • macOS 26.5, Apple Silicon:
    • scripts/test-local.sh --jobs 4 ran the full suite: 6,352 of 6,355 passed in 32 minutes. The three failures were the grep warning.
    • After the fix, those suites plus test_fix_reviews.bats (340 tests) and the new preflight test: 471 of 471 passed.
  • Stock bats tests/test_ci_status.bats now stops in setup_suite with a message naming bash 3.2, BSD sed, BSD grep, BSD date and BSD awk.
  • /bin/bash scripts/test-local.sh … (bash 3.2) works. With a fake HOMEBREW_PREFIX it exits 1 and names the missing formulae.
  • shellcheck (warning), check-bats-negation.sh, check-fixed-tmp-paths.sh and markdownlint all pass.
  • Not verified on Linux locally; CI on this PR covers that.

🤖 Generated with Claude Code

Problem

No linked issue: see the Why section above (2,039 failing tests on stock macOS, 160 with the existing PATH workaround, 0 with this PR).

Risk

Low — changes automation shell logic under scripts/, covered by shellcheck (--severity=warning) and the bats suite.

Test plan

Tests added/updated: tests/dev-lead/unit/setup_suite.bash,tests/dev-lead/unit/test_toolchain_preflight.bats tests/dev-lead/unit/toolchain_preflight.bash. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

This PR's Lint (shellcheck) and bats checks show pass/fail; watch subsequent dev-lead / pr-review runs for behavioral regressions.