docs(benchmarking): default to 3 runs per arm, up to 5

Merged
BO
1d ago
+28-9

hr-benchmarking now has a default stopping rule: 3 runs per arm, plus up to 2 more per arm only when the arms' ranges overlap or the effect is near the noise, so 5 per arm is the cap.

Before, the skill said repetition counts "depend on the decision and observed variation; there is no universal minimum" and asked only that a count be declared first. Nothing stopped an agent from choosing 10 or 12 runs for a result that 3 would settle, and references/javascript.md asked for "at least 10 runs" to time startup.

What changes:

  • Default: 3 alternating runs per arm; 2 more per arm only on overlapping ranges or an effect near the noise; cap 5. If 5 leave it unresolved, the result is inconclusive. Another count needs a reason declared before the first run.
  • Results give the run count and range with any central value, never a bare average.
  • The benchmark's expected wall-clock time is stated before the first run.
  • references/javascript.md no longer asks for at least 10 runs; it points at the default. The 11- and 12-run figures there are recorded past measurements and stay.

Kept as before: the A/A arm or already-known variation, the variation criterion set before the first run, retained results, pilots kept separate from the claim's sample, and the absolute floor for short or tail metrics.

Agent behavior it changes: a benchmarking agent plans 3 runs per arm instead of choosing its own count. I checked the wording by reading it against the rest of the skill and its references. I ran no model eval; the repository has none for this skill, and the description (which evals/skill-descriptions covers) is unchanged.

Other places with run counts: hr-writing-for-agents and evals/skill-fires say "at least three runs per arm", which agrees. hr-agent-guidance-audit/references/prune-workflow.md asks for 4 to 9 pass/fail runs per arm to test a rule; it is a different decision and is left alone here.

Checks: node --test test/*.test.mjs (151 pass, 7 skipped), bun test skills/ (335 pass), npm run check:package -- --bun (234 files), pinned oxfmt --check and oxlint clean. CHANGELOG.md has an Unreleased entry. Not landed; the maintainer decides.

Summary by cubic

hr-benchmarking now sets a default stopping rule for performance claims: 3 alternating runs per arm, 2 more per arm only when the arms' ranges overlap or the effect is within the variation criterion, and a cap of 5. If 5 runs leave the comparison unresolved, the result is reported as inconclusive. A different count is used only when the user asks for it or a run is too long to repeat 3 times, and must be declared with a reason before the first run.

Changed behavior

  • Results must give the run count and range with any central value, never a bare average.
  • The expected wall-clock time is stated before the first run.
  • references/javascript.md no longer asks for at least 10 runs when timing startup; it points to the default. The 11- and 12-run figures there remain recorded past measurements.

Summary by CodeRabbit

  • Documentation
    • Updated benchmarking guidance to use three alternating runs per arm by default, adding two only when results overlap or meet the preset variation criterion, up to five runs.
    • Results that remain unresolved after five runs are reported as inconclusive. Alternative run counts require a reason declared before testing.
    • Reports include run counts and ranges alongside central values, and state expected elapsed time before testing begins.
    • JavaScript startup benchmarks no longer require at least ten runs.