test(verify): add a project verify skill and document it in AGENTS.md

Merged
HA
8d ago
+930-9

Summary

Until now an agent could only prove a change with pnpm verify:local, which runs everything in one go. This PR adds a project skill at .claude/skills/verify/ so agents can work and verify on their own:

  • Change-to-gate map. SKILL.md lists, for each source area, the checks to run while iterating and the gate that counts as completion evidence.
  • scripts/stack.sh. It keeps a disposable Postgres/Redis/API stack running between calls. It reuses infra/docker-compose.verify.yml with a unique Compose project, random loopback ports, a clean env -i, and migrations on startup. Subcommands:
    • up, restart, down: start, rebuild, and stop the stack. down stops only the PID it started and keeps the evidence.
    • doctor: checks service health, PID and port ownership, readiness, and token auth. It fails when dist/ is older than src/ or when node_modules doesn't match pnpm-lock.yaml.
    • api, mcp, psql: send an authenticated request, call one MCP tool, or inspect the database.
    • suite <name>: runs any scripts/verify/*.mjs suite, or integration [filter], against the stack.
  • Feature map. features/*.md covers auth, sandboxes, exec, files, ingest, and MCP. scripts/journeys.sh runs those journeys with 41 assertions in about 8 seconds.
  • Redacted evidence. scripts/verify/.run/stack-<id>/transcript.jsonl masks the stack secret, JWTs, and token, secret, password, and authorization keys.
  • AGENTS.md (and CLAUDE.md, which links to it) gets a new "Verify while you work" section with the commands and a table of the scripts/verify suites. Agents without Claude Code can run the scripts directly.

No product code changes; nothing in src/ is touched. The one CI-gate edit: scripts/verify-local.sh now calls scripts/verify/assert-integration-report.mjs instead of an inline heredoc. It runs the same three assertions with the same messages, and stack.sh suite integration calls the same file. Node tests cover empty, failed, and skipped reports.

Test plan

Run in this worktree on top of e9cf495:

  • pnpm check passes: 2334 unit tests and 154 TypeScript SDK tests.
  • Fresh up, then doctor (all ok), all 6 journeys (41/41), then down. No containers, volumes, listeners, or server process remain, and the evidence directory survives.
  • Each scripts/verify suite passes through stack.sh suite: contracts, output-cap, shutdown, sdk-clients, worker-failures, sse-finalization, input-budgets, tenants, ingest-admission, and integration (265 tests, 0 skipped). large-files was not run because it needs a cgroup v2 host.
  • Negative control: changing a read-only error code in src/api/routes/exec.ts made doctor report the stale build and journeys.sh exec fail with the expected and actual values. Reverting the change made it pass again.
  • Failure paths:
    • a failed launch prints the server log tail, blocks a second up, and down still cleans up;
    • dependency drift fails doctor;
    • an integration report with skipped tests fails the gate.
  • Redaction: the bootstrap call's X-Auth-Secret and the returned token are recorded as [REDACTED]. A scan found no JWTs or secrets in the transcript.
  • pnpm verify:architecture passed while a stack was up, so the two coexist.
  • With the shared integration check: pnpm check:scripts (including the new tests), pnpm lint, and pnpm verify:local all pass (265 integration tests, 0 skipped). On the stack, suite integration passes on a real report and fails on the same report with one skipped test.

Follow-ups (not in this PR)

  • MCP bash_exec reports denial as {"exitCode":1,"stderr":"forbidden"}, and file and sandbox tools report it as {"ok":false}. Both use isError:false. Setting isError:true (in src/api/mcp/tool-errors.ts) would match the MCP spec, but it is a contract change.
  • stack.sh repeats about 15 lines of compose and env setup from verify-local.sh.

🤖 Generated with Claude Code

Summary by cubic

Adds a verify project skill so agents can prove changes against a live, disposable Postgres/Redis/API stack instead of only running the full pnpm verify:local gate.

New Features

  • scripts/stack.sh keeps a disposable Postgres/Redis/API stack running between calls, with up, restart, doctor, api, mcp, psql, suite, and down subcommands.
  • SKILL.md maps each source area to the checks to run while iterating and the gate that counts as completion evidence.
  • features/*.md and scripts/journeys.sh drive client-visible journeys for auth, sandboxes, exec, files, ingest, and MCP with about 40 assertions in roughly 10 seconds.
  • API/MCP calls are recorded to transcript.jsonl; bodies are capped at 4 KiB and the stack secret, JWTs, and sensitive keys and header values (including inside JSON) are redacted.
  • AGENTS.md documents the workflow; no product code changes, and existing gates like verify:local can run alongside the stack.
  • The integration report gate, extracted from verify-local.sh, is now shared as scripts/verify/assert-integration-report.mjs with node tests.