feat(review): opt-in AI verifier that confirms or refutes each AI comment from dbt evidence
Issue for this PR
Part of #1792
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
Stacked on #1758. Adds an opt-in verifier for the AI review lane (ALTIMATE_REVIEW_AI_VERIFY=1). After the AI reviewer drafts its comments, one more bounded model pass tries to confirm or refute each one from dbt evidence. Comments it can't support are moved to the summary or dropped. Off by default: with the flag unset, the review output is unchanged.
Results (replay of public dbt PRs, blind Claude judge)
Verifier on vs off, same runs (50 PRs, 59 reviewer-found defects, tuning repos)
| AI comments per PR | Useful | Wrong | |
|---|---|---|---|
| Verifier off | 2.48 | 45% | 1% |
| Verifier on | 1.32 | 53% | 0% |
- On 26 PRs from repos never used for tuning: useful 38% → 59%, wrong 1 → 0.
- The first version dropped 3 real catches by discarding cross-file claims it could not check. Those claims now go to the summary instead of being dropped (
3b2c31e2dd), so catches are not lost.
Repos never used for any tuning (25 repos, 101 PRs, one snapshot each, 153 defects)
| Build | Defects caught | AI comments useful / wrong |
|---|---|---|
| Released v0.12.11 with its AI | 24 | 34% / 3.6% |
| This stack, AI + verifier + rules | 35 | 43% / 1.3% |
Difference +11 catches (95% CI +1 to +22, PR-level bootstrap). The verifier costs about 1.4× the tokens of the AI review alone.
What changed
ai-verify.ts:- Deterministic features: evidence line valid, targets unchanged code, restates an engine finding, claims another file, style only, low confidence, hedged data-diff.
- The verifier prompt and a tolerant parser.
- A bounded tool loop:
read_file,compiled_sql,model_columns,column_reach,model_config,downstream,equivalence,grain,grep_project,pr_description. At most 3 rounds and 12k tokens. - Table-driven triage into keep / summary / discard.
- Fails open: on an error, timeout or missing verdict, the original findings stay.
- Verifier decisions, tokens and dropped findings are recorded in the envelope and telemetry.
ALTIMATE_REVIEW_AI_REPLAY=<envelope>replays stored generator findings so only the verifier is re-run.- Runner gains
columnReach, and the stream reader keeps tool parts.
How did you verify your code works?
bun test test/altimate/review: 667 pass.bunx tsc --noEmit: clean.- New:
test/altimate/review-ai-verify.test.ts(features, every triage row and knob, tool loop limits, allowlist, fail-open, parser).
Cost side effects
None by default: the verifier is opt-in (ALTIMATE_REVIEW_AI_VERIFY=1). When enabled it adds one bounded model pass per review (at most 3 tool rounds and 12k tokens), measured at about 1.4x the AI-lane tokens of the AI review alone. Its tools read local project files, the manifest and the in-process engine; no warehouse queries. Verifier tokens are recorded in the envelope and telemetry.
Screenshots / recordings
Not a UI change.
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
- I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
🤖 Generated with Claude Code
Summary by cubic
Adds an opt-in, dbt-grounded verifier for the AI review lane. With ALTIMATE_REVIEW_AI_VERIFY=1, one bounded model pass after the AI reviewer tries to confirm or refute each comment from dbt evidence (compiled SQL, DAG lineage, project grep); comments it can't support move to the summary or are dropped. With the flag unset the review output is unchanged. The verifier fails open: any error, timeout, or exhausted budget leaves the findings it didn't review untouched.
Results (blind judge on public dbt PRs)
| AI comments per PR (% useful / % wrong) | |
|---|---|
| Verifier off | 2.48 (45% / 1%) |
| Verifier on | 1.32 (53% / 0%) |
On 26 PRs from repos never used for tuning, useful went 38% → 59% and wrong went 1 → 0. Cross-file claims the verifier can't check go to the summary instead of being dropped, so real catches aren't lost. The verifier costs about 1.4× the tokens of the AI review alone.
What changed
- Deterministic features per finding (evidence line valid, unchanged code, engine grounding, cross-file claim, style only, low confidence), a tolerant verdict parser, and table-driven triage into keep / summary / discard.
- A bounded tool loop of 10 read-only tools, at most 3 rounds and 12k tokens.
- Verifier decisions, tokens, and dropped findings are recorded in the envelope and telemetry;
retriage()re-runs triage offline without a model call. ALTIMATE_REVIEW_AI_REPLAY=<envelope>replays stored generator findings so verifier settings can be compared without regenerating.grepProjectandcolumnReachare wired into the runner for evidence gathering.- Coverage: 667 tests pass, including new suites for features, every triage row and knob, tool loop limits, fail-open, and replay.