Compare two builds over paired bench rounds
Intent: Give the benchmark comparison a mode that measures a base build and a head build in the same job, so a change is judged against the commit it builds on rather than against tests/bench-baseline.json. This is the first of the four pull requests in #503; nothing calls the new mode yet.
This PR: scripts/bench-compare.py --paired --base b1 ... bN --head h1 ... hN takes one results file per round and pairs the Nth base file with the Nth head file. For each metric it takes the median of the head/base ratios and a bootstrap 95% interval of that median (2000 resamples, fixed seed). A metric is slower or faster only when the interval excludes 1 and the median is past --threshold; otherwise it is "no significant change". A head value lost in any round gives "lost on head", because the head build failed where the base did not; a base-only loss only drops that pair, and both tables show the lost rounds per side. A metric with fewer than 5 usable pairs is "too few pairs", because with one pair the interval is a single point and always excludes 1. A metric with no usable pair is "no result", and applications.make_ms is shown without a verdict, because its samples jump between two levels inside one run. The Base and Head columns are labelled as per-side medians, since the change is the median of the per-pair ratios. Paired mode exits 0 on valid input, and 2 on more than 32 rounds, rounds with no metric, a report over 64 KiB, or the static flags --results, --baseline and --fail-on-missing. --report-json writes the per-round values (schema 3), not verdicts; scripts/bench-report.py validates that report and recomputes each verdict with the same paired_verdict code, so the comment and the job log agree. Only slower, faster and lost on head make the reporter print "changes"; a metric missing on both sides stays clean, because the head build did not cause it. A #498 lat_fs timeout in one head round therefore posts a comment.
Paired mode has no default threshold. #503 sets it from A/A runs on main, which can start only after the workflow lands, so with neither --threshold nor BENCH_REGRESSION_THRESHOLD both scripts show every ratio and interval with "no threshold" instead of a verdict. The reporter reads BENCH_REGRESSION_THRESHOLD only for a schema 3 report; schema 2 reports and the static mode keep 0.20.
The first commit stands on its own: a lone surrogate in a metric name passed the name check and then crashed the reporter while it wrote the comment, in upstream's schema 2 path too. The name check now rejects it.
Rebased on upstream/main at 21295ee. Commands and results:
make check-bench-reportpasses at each commit. Its new cases cover each verdict (steady slowdown and speedup, noise around 1, a median past the threshold with an interval across 1 on either side, a change inside the threshold, a base-only loss, a head-only loss, too few pairs, a metric missing on both sides,make_ms, a zero base, no threshold), malformed and empty schema 3 reports, surrogate names in both schemas, null values, an unreadable round file, more than 32 rounds, rounds with no metric, a report over 64 KiB, the three static flags with--paired, a schema 2 report keeping 20% withBENCH_REGRESSION_THRESHOLDset, andbench-compare.py --paired --report-jsonruns at 32 rounds read back and rendered by the reporter with the threshold from the environment and with none.- Each of fourteen edits to that code makes the self-test fail, among them dropping the head-loss check, leaving "lost on head" out of the listed verdicts, dropping the interval check on either side, setting the pair minimum to 1, dropping the
make_msexemption, accepting surrogates, and falling back to 0.20 for a schema 3 report. - The static mode prints byte-identical output with the same exit code as upstream/main's
bench-compare.pyon two results files, with the default threshold,--threshold 0.05,--fail-on-missingand--report-only. make checkpasses.
Not tested: the 5-pair minimum and the interval on real CI noise, and the comment job that renders the report. Both come with the workflow in the next pull request; A/A runs on main after it lands set the threshold.
Refs #503
Summary by cubic
Adds a paired benchmark comparison mode so a pull request is judged against the commit it merges into rather than the checked-in baseline. Nothing calls the new mode yet; this is the first of four pull requests in #503.
bench-compare.py --paired --base b1 ... --head h1 ... pairs the Nth base file with the Nth head file. Per metric it prints the median of head/base ratios with a bootstrap 95% interval (2000 resamples, fixed seed). A metric is slower or faster only when the interval excludes 1 and the median is past --threshold; fewer than 5 usable pairs give "too few pairs", no usable pair gives "no result", and applications.make_ms is shown without a verdict because its samples jump between levels inside one run. Paired mode has no default threshold: with neither --threshold nor BENCH_REGRESSION_THRESHOLD, every such metric shows no threshold with its ratio and interval. Paired mode exits 0 on valid input; an unreadable round file or one with no usable data exits 2, and it caps at 32 rounds per side, matching what the reporter accepts.
--report-json writes the per-round values as a schema 3 report, which bench-report.py validates as untrusted input and re-judges with the same paired_verdict code, keeping the job log and the comment consistent. The reporter's --threshold defaults to BENCH_REGRESSION_THRESHOLD like bench-compare.py's. Only slower, faster, and lost on head post a new comment; "too few pairs" and a metric missing on both sides stay clean. Paired mode exits 2 before writing when a round holds more than 256 metrics, a metric name over 160 characters, or a name with a control, format, or surrogate character, which the reporter would reject; the surrogate part of that check covers schema 2 too. Unreadable results files now exit 2 in both modes instead of ending in a traceback. Schema 2 reports and the static baseline comparison are unchanged.