docs(skills): bug bash repros from the finding, charters sized to the clock

Merged
OK
8d ago
+161-69

The bundled skill's bug bash had a coding agent write every repro test with exact locators, found one by one through e2e mcp locate, before a finding counted. That is the most expensive part of a bash and the one most likely to fail for reasons unrelated to the bug. Its charters also stopped at --max-steps 6, a number from when explore capped runs at 12 steps.

  • Repros from the finding. A repro test is the finding's reproduction as agent.act steps, one goal each, plus its expected behavior as the closing agent.assert. It needs no locators, and the charter's credential goes in as step parameters (or a setup test's session).
  • Confirmed means the repro fails on the reported defect: ASSERTION_FAILED from the closing assert, or a failed act whose message names the defect. A failed act carries the code the model chose, so the skill says to read its message and screenshot. A pass means the bug did not reproduce.
  • Assert only the defect. Repros share the app's data, so a count another repro changes fails for the wrong reason.
  • Verifiers stay apart. Each subagent runs its own area's repros with the bug-bash config into --output .e2e/bugbash/verify-<area>, since a run clears its output's results. The report runs every repro once with npx e2e run --tag bugbash --config e2e.bugbash.config.ts.
  • Scripted hardening (screen actions, exact expect, MCP locate) is a step for when the user wants a regression test. A scripted step that opens a native dialog needs browser.onDialog.
  • Charters sized to the clock: --max-steps 15 --timeout 900000, about fifteen minutes. Under a time limit the first round ends by a third of it, the bash keeps running charters for what no charter reached until a tenth is left, and each bug is reported once its repro confirms it.
  • An account the app seeds with a password under 6 characters (admin/admin) is called out before the config fails on it.
  • docs/bug-bash.mdx, docs/reference/mcp.mdx, and the create-verification-skill generator and its example now describe the same bar.
// tests/bugbash/cart-line-quantity.e2e.ts
import { test } from '@e2e-dev/web';
import { credentials } from 'e2e';

test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => {
  const shopper = credentials.user('bb-cart');
  await app.open('/');
  await agent.act('sign in with the given credentials', {
    params: { username: shopper.username, password: shopper.password },
  });
  await agent.assert('the app shows you are signed in');
  await agent.act('add "Dune" to the cart');
  await agent.act('set the quantity of "Dune" in the cart to 2');
  await agent.assert('the cart line for "Dune" totals twice its unit price');
});

Verified

Ran it locally: yes

  • The example above typechecks in the testbed (tsc --noEmit).
  • Agentic repros written this way for a scratch notes app's findings, npx e2e run --tag bugbash with openai/gpt-6-luna-fast: two real bugs failed with ASSERTION_FAILED naming the defect, one from the closing assert and one from the act that met it. The not-a-bug control passed. The three repros took 14.9 s and $0.0032 together.
  • A first draft of the control asserted a note count. It failed because another repro had deleted notes from the shared app, which is why the skill now says to assert only the defect.
  • With #1599, the same repros get past confirm() and leave-page dialogs. Without it, the confirm-gated repro ends AUTOMATION_UNSUPPORTED and the control ends ACTION_FAILED.