feat: re-run evidence in `answer_commit`: vacuous filters, reading comparison, alternative columns

Draft
AN
3d ago
+825-1

Issue for this PR

None (new feature; no linked issue).

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Wrong analyst answers are usually a misread question, not broken SQL. Two common forms: a filter on the wrong column, which excludes nothing, and two readings of an ambiguous clause where the agent picks one by wording. answer_commit is the last point where the agent can still change its answer. A note added after the commit does nothing, because the agent repeats the committed text unchanged.

Before it accepts an answer, answer_commit now re-checks it against data and rejects it once per turn when the data contradicts it:

  • Vacuous filters. It takes the turn's last sql_execute query and re-counts its rows with each top-level WHERE conjunct removed in turn (at most 6). If removing a conjunct leaves the count unchanged, that conjunct is reported. IS NOT NULL guards are skipped.
  • Reading ledger. readings entries can now carry sql and warehouse. When two or more readings carry SQL, answer_commit runs each one and compares the results as unordered row sets with typed values, at most 200 rows.
    • If the results disagree and the commit has no reading_evidence, the commit is rejected.
    • If they agree, that is recorded in metadata.
  • Schema-derived alternatives. When there is a finding, the review also lists other date-like or status-like columns of the same tables. These come from the catalog only.

The review is a status: "error" result marked answer_commit review:. The next commit in the same turn is accepted. It is meant as a second look, not a gate. Commits with nothing to re-run stay synchronous.

Behavior: was → now

  • Commit after a local DuckDB query with a zero-effect filter: was accepted at once → now rejected once with the filter named, then accepted.
  • Readings with SQL that return different results: was accepted → now rejected once unless reading_evidence is given.
  • Everything else, including remote warehouses, SQLite, Postgres and checks turned off: no change.

New schema fields, all optional: readings[].sql, readings[].warehouse, reading_evidence.

Cost side effects

  • Where it runs. Local DuckDB only: the session scratch database, or a DuckDB file connection whose path is local. Remote warehouses are never queried, and attached non-DuckDB or remote databases are rejected.

  • What it runs. A single plain SELECT with:

    • no CTEs, subqueries or table functions;
    • only built-in functions, so user macros are rejected;
    • joins only with a column equality.

    Every base table must have at most 500,000 rows by catalog estimate, and no view may share its name.

  • How much. Counts go through a subquery capped at 1,000,001 rows; a larger count skips the check. Each check stops sending queries after 250 ms.

    • Vacuous-filter check: at most 1 + 6 counting queries plus catalog lookups.
    • Readings: one capped probe plus one run per reading.

  • Interruption. Running queries are never interrupted, because interrupting restarts the local engine and drops scratch tables. The size gates above are what bound the work.

  • No model calls. The check adds no tokens beyond the one review message.

  • Off switch. ALTIMATE_SQL_AUTO_CHECK=off (or 0 / false) disables all of it.

  • Known limit. The per-row cost of built-in scalar functions is not bounded. A reading that calls one with a huge argument could run until the driver's query timeout, which is 120 s by default. The same SQL run through sql_execute would have the same cost.

How did you verify your code works?

All of this runs against real DuckDB (session scratch) and SQLite fixtures, with no mocks of the engine.

  • test/altimate/tools/answer-commit-evidence.test.ts (17 tests) covers:

    • a zero-row filter is flagged; effective filters and IS NOT NULL guards are not;
    • these are skipped: SQLite, a 600k-row table, subqueries, CTEs, comma joins, a non-equality join, an equi-join past the row cap, user macros;
    • an inner equi-join is checked;
    • readings that agree are recorded; readings that disagree get one rejection and are accepted with reading_evidence;
    • NULL vs 'null' and 1 vs '1' are kept distinct;
    • the alternative-columns line;
    • the tool rejects once, then accepts the recommit;
    • the check reads only the committing session's scratch store;
    • ALTIMATE_SQL_AUTO_CHECK=off does nothing.

  • These suites also pass: test/altimate/tools/answer-commit* (230 tests), plus tool-schema*, sql-execute* and test/agent (579 tests, 0 fail). bun run typecheck is clean. script/upstream/analyze.ts --markers --base main --strict is clean.

  • Two rounds of local read-only Codex review. The first round's findings are fixed:

    • the scratch store was resolved from the wrong session;
    • joins were unbounded;
    • the budget did not stop later queries;
    • a CTE name could hide a table from the size gate;
    • values were compared as untyped strings.

    The second round's findings are also fixed: nested-loop joins and user macros are now rejected. The remaining finding is the driver-timeout note under the known limit above. Reviewed head: 3eb91df05b.

  • Not run: an end-to-end agent session. This PR adds no evaluation runs.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Changes the analyst terminal commit path and executes extra read-only SQL on local DuckDB; bounded and opt-out, but a mistaken rejection or session-scoping bug could block or skip checks incorrectly.

Overview
answer_commit now runs optional, bounded local DuckDB checks before accepting an analyst answer (toggle ALTIMATE_SQL_AUTO_CHECK=off). When evidence contradicts the commit, the tool returns status: "error" once per turn with an answer_commit review: message; a second call with the same answer proceeds (or reading_evidence when readings disagree).

New module answer-commit-evidence re-runs the turn’s last sql_execute query under strict limits (~250 ms, row caps, plain SELECTs only): vacuous filters (WHERE conjuncts that remove no rows), reading comparison when two+ readings[].sql disagree, and catalog hints for alternate date/status-like columns. Evidence is attached to commit metadata on success.

Schema additions: readings[].sql, readings[].warehouse, reading_evidence. Integration tests cover DuckDB scratch session scoping and skip paths (SQLite, large tables, unsafe SQL).

Reviewed by Cursor Bugbot for commit 3eb91df05b4c28dd563efe3c6087b61ff79951be. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

answer_commit now re-checks the committed answer against data on local DuckDB before accepting it, rejecting the commit once per turn when the evidence contradicts it comma (previously commits were accepted immediately). The check is bounded and optional, so it stays synchronous and safe by default.

Behavior

  • Commits with a vacuous WHERE filter (one that removes no rows) are rejected once, with the filter named.
  • Readings that carry sql are executed and compared as typed, unordered row sets; disagreement rejects once unless reading_evidence is provided.
  • Agreeing readings and vacuous-filter findings are recorded in commit metadata.
  • The review also lists other date-like or status-like columns of the same tables.
  • The check runs only on local DuckDB (scratch or local file), with a 250 ms budget, a 500k-row table ceiling, capped counts, no user macros or non-equi-joins, and no query interruption.
  • ALTIMATE_SQL_AUTO_CHECK=off (or 0 / false) disables it; everything else stays unchanged.

Summary by CodeRabbit

  • New Features
    • Answer submissions can now be checked against local query results for filters that do not affect results and disagreements between readings.
    • When a review is needed, the submission is paused with supporting evidence and can be retried after clarification. Agreeing readings may include evidence in the committed result.
    • Automatic checks are limited to eligible local DuckDB queries and can be turned off.