Confirm benchmark verdicts and run A/A nightly
Intent: Report a slower or faster benchmark verdict only when a second set of pairs gives it again, let a reviewer measure a pull request again by adding the bench label, and measure main on a schedule: nightly A/A, and weekly against the main of 4 weeks earlier. This is PR 3 of #503, without the two group thresholds.
This PR:
tests/bench-suite.shtakesBENCH_METRICS, the metrics to run.scripts/bench-paired.shmeasures 6 more pairs of the flagged metrics and adds them to the report;bench-compare.pyandbench-report.pykeep a verdict only when that second set gives it again; otherwise it readsnot confirmed, orlost on headwhen the second set's head lost a value, so abench-compare.py --pairedrun without a second set gives noslowerorfaster. A metric is flagged when it readsslowerorfasteratBENCH_FLAG_THRESHOLD: 0 for apull_requestrun, which from a fork may not get the repository variable (the current GitHub docs on variables do not say), for an A/A run, whose second set is threshold data, and for the weekly run;report, the threshold of the verdicts, for the other runs, so while the repository variable is unset they measure no second set. The second set is best effort: when a round of it or its comparison fails, or it stops before its first pair, the report says so and Tier 3 still runs; a run that posts no comment shows that only in its job summary.- The second set is cut at
BENCH_CONFIRM_DEADLINE(1080 s;docs/testing.mdhas the exact rule), so Tier 3 still fits in the 25 minute step. On the first CI run that left room only for pairs of the flagged metrics under about 51 s each, calibration readings included; when a pair takes longer the set is cut, and a metric that readsslowerat the configured threshold readsnot confirmed. benchmark.ymladds thebenchlabel trigger and the twoschedulelines. Two choices differ from #503: the weekly base comes fromgit rev-list -1 --first-parent --before='4 weeks ago', because without--first-parentit can be a commit of a branch merged later, and schedule runs keep their artifacts 90 days instead of 14, so the A/A data is still there when the thresholds are set from it.bench-report.ymlcomments on a labeled pull request even when the report is clean, then takes the label off (also when it leaves the comment to a later merge ref's report), so adding it again measures again, and it ignores a run whose jobs were all skipped.
If the two sets were independent, an unchanged metric would pass both about 0.05% of the time (2 x (1/64)^2), not the 0.1% #503 gives; the nightly A/A data will show the real rate.
Left out: the two group thresholds (they need A/A data, so BENCH_REGRESSION_THRESHOLD stays unset), the runs against a deliberately slowed branch, and creating the bench label.
Testing:
make check-bench-reportpasses under Python 3 and 3.9 at each commit;make check-formatandscripts/check-commit-log.shpass.- Each new self-test and stub-test case fails when the code it covers is broken.
- End to end, a head 50 ms slower at startup read
slowerin both sets (5.06x, then 5.11x).
Not tested: No test runs three BENCH_METRICS paths of tests/bench-suite.sh, because they need an elfuse build: a tier left out and the lat_fs drop were checked on local runs, and ERR in place of the no-results stop was not run. On GitHub Actions only the opened path has run (run 44875161536, nothing flagged), and on an earlier revision (939253c), so the scope step's choice of BENCH_FLAG_THRESHOLD and the best-effort second set have not run there. The bench label does not exist yet, the schedule lines run only on main, and bench-report.yml runs from main, so those changes, the label removal among them, are checked by reading. The second set's timing on CI.
Refs #503