Confirm benchmark verdicts and run A/A nightly

XA
4m ago
+1184-178

Intent: Report a slower or faster benchmark verdict only when a second set of pairs gives it again, let a reviewer measure a pull request again by adding the bench label, and measure main on a schedule: nightly A/A, and weekly against the main of 4 weeks earlier. This is PR 3 of #503, without the two group thresholds.

This PR:

  • tests/bench-suite.sh takes BENCH_METRICS, the metrics to run.
  • scripts/bench-paired.sh measures 6 more pairs of the flagged metrics and adds them to the report; bench-compare.py and bench-report.py keep a verdict only when that second set gives it again; otherwise it reads not confirmed, or lost on head when the second set's head lost a value, so a bench-compare.py --paired run without a second set gives no slower or faster. A metric is flagged when it reads slower or faster at BENCH_FLAG_THRESHOLD: 0 for a pull_request run, which from a fork may not get the repository variable (the current GitHub docs on variables do not say), for an A/A run, whose second set is threshold data, and for the weekly run; report, the threshold of the verdicts, for the other runs, so while the repository variable is unset they measure no second set. The second set is best effort: when a round of it or its comparison fails, or it stops before its first pair, the report says so and Tier 3 still runs; a run that posts no comment shows that only in its job summary.
  • The second set is cut at BENCH_CONFIRM_DEADLINE (1080 s; docs/testing.md has the exact rule), so Tier 3 still fits in the 25 minute step. On the first CI run that left room only for pairs of the flagged metrics under about 51 s each, calibration readings included; when a pair takes longer the set is cut, and a metric that reads slower at the configured threshold reads not confirmed.
  • benchmark.yml adds the bench label trigger and the two schedule lines. Two choices differ from #503: the weekly base comes from git rev-list -1 --first-parent --before='4 weeks ago', because without --first-parent it can be a commit of a branch merged later, and schedule runs keep their artifacts 90 days instead of 14, so the A/A data is still there when the thresholds are set from it. bench-report.yml comments on a labeled pull request even when the report is clean, then takes the label off (also when it leaves the comment to a later merge ref's report), so adding it again measures again, and it ignores a run whose jobs were all skipped.

If the two sets were independent, an unchanged metric would pass both about 0.05% of the time (2 x (1/64)^2), not the 0.1% #503 gives; the nightly A/A data will show the real rate.

Left out: the two group thresholds (they need A/A data, so BENCH_REGRESSION_THRESHOLD stays unset), the runs against a deliberately slowed branch, and creating the bench label.

Testing:

  • make check-bench-report passes under Python 3 and 3.9 at each commit; make check-format and scripts/check-commit-log.sh pass.
  • Each new self-test and stub-test case fails when the code it covers is broken.
  • End to end, a head 50 ms slower at startup read slower in both sets (5.06x, then 5.11x).

Not tested: No test runs three BENCH_METRICS paths of tests/bench-suite.sh, because they need an elfuse build: a tier left out and the lat_fs drop were checked on local runs, and ERR in place of the no-results stop was not run. On GitHub Actions only the opened path has run (run 44875161536, nothing flagged), and on an earlier revision (939253c), so the scope step's choice of BENCH_FLAG_THRESHOLD and the best-effort second set have not run there. The bench label does not exist yet, the schedule lines run only on main, and bench-report.yml runs from main, so those changes, the label removal among them, are checked by reading. The second set's timing on CI.

Refs #503