docs(agent-guidance-audit): call the pass/fail run count a deliberate exception

Merged
BO
3d ago
+9-3

hr-agent-guidance-audit's prune workflow now says its pass/fail screening count is a deliberate exception to the 3-runs-per-arm default in hr-benchmarking.

The workflow tells the agent to use --runs 4 or more, and 9 runs per arm for a smaller gap. The hr-benchmarking default (3 runs per arm, up to 5) is for performance claims, and read next to this line it looked like a contradiction. A count of passes separates arms far less finely than a timing range does: 4 runs separate a rule only when the arms differ by 4 and 0, and 8 of 9 against 3 of 9 gives p = 0.05.

The count and its reasoning are unchanged. The edit adds one sentence naming the exception and pointing at the default. A CHANGELOG.md entry is under Unreleased.

Checks: pinned oxfmt --check and oxlint clean; the reference-link tests pass (20). The full test suites were not run because the host was reserved for another job; the change is one paragraph of Markdown. Not landed; the maintainer decides. The branch is rebased on main after #319 landed.

Summary by CodeRabbit

  • Documentation
    • Clarified that pass/fail screening uses a higher run-count threshold than performance comparisons: at least 4 runs, or 9 runs per arm when the gap is smaller.
    • The existing thresholds remain unchanged. Guidance distinguishes these screening thresholds from the 3-runs-per-arm default for performance claims, making the applicable run counts clearer.