feat(review): opt-in test-gap check and focused AI batches; verifier reads every changed file

RA
30m ago
+1249-25

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1771. Adds two opt-in AI-lane options, lets the verifier read every changed file, and fixes the rule gaps the planted-bug suite found. Defaults are unchanged.

What changed

  • ALTIMATE_REVIEW_AI_TEST_GAP=1: when a changed macro, materialization or generic test changes runtime behaviour, the AI says whether the PR adds a test that runs it, as one test_coverage comment. Each batch also gets the PR's other changed paths.
  • ALTIMATE_REVIEW_AI_FOCUSED=1: on PRs with 7 or more files, related files are reviewed together in batches of up to 5. Related means a macro with its dispatch variants, a test with the model it refs, and a yml with the models it declares. Grounding is filtered per batch, findings are deduplicated, and there are at most 4 batches. No paid measurement yet, so it stays opt-in.
  • The verifier can read every file the PR changes, so it can confirm "no test was added".
  • Rule fixes from the planted-bug suite:
    • three recall rules;
    • the join chain across a join with no ON clause;
    • parse-breaking findings are kept in dev, sandbox and scratch folders.

Results

Test-gap check

  • A private 5-project monorepo PR that changed a materialization with no test: the AI raised the missing test in 10 of 10 runs, against 2 of 21 before.
  • 10 tuning-repo PRs: catches unchanged (5 of 16), useful comments 51% → 57%, wrong 3% → 0% (blind Claude judge), tokens 1.02×.

Planted-bug suite on repos never used for tuning: 236 of 238 caught, 1 false alarm on 237 no-defect twins.

How did you verify your code works?

  • bun test test/altimate/review: 748 pass.
  • bunx tsc --noEmit: clean.
  • New: test/altimate/review-ai-test-gap.test.ts, plus focused-batch cases in review-ai-inputs.test.ts.

Cost side effects

None by default; both options are opt-in. ALTIMATE_REVIEW_AI_TEST_GAP=1 measured at about 1.02x AI-lane tokens. ALTIMATE_REVIEW_AI_FOCUSED=1 splits PRs with 7 or more files into at most 4 batches, so up to 4 AI calls per review instead of one (not yet measured, hence opt-in). Letting the verifier read every changed file only applies when the verifier is enabled and stays within its 12k-token cap. No warehouse queries or cloud resources.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code