Add a three-tier benchmark suite

XA
9m ago
+9490-22

Supersedes #223, based on the benchmark suite @Max042004 wrote there.

make bench runs tests/bench-suite.sh in three tiers and writes build/bench-results.json:

  • Tier 1: lmbench lat_syscall, lat_proc and lat_fs, built from a pinned source snapshot by tests/fetch-fixtures.sh.
  • Tier 2: application workloads over the Alpine fixture tools (startup, python startup, git status, ripgrep, zstd, make over tests/bench-corpus).
  • Tier 3, new here: the python, node, C compile and JVM rows of #224, each run in an OCI image pinned by digest through scripts/elfuse-oci.py. Image pulls happen before the timed samples. BENCH_OCI=0 skips the tier. The go row is left out: gofmt there can need a 65th thread, and elfuse gives a guest process 64 (#494).

The Tier 2 tools land in the shared fixture rootfs, so every qemu boot carries them: the initramfs grows from 35.4 MB to 51.8 MB gzip (61 MB to 107 MB unpacked, out of the 2048 MB guest), and qemu boot to ssh stays at 1.1 to 1.2 s over 3 boots each.

The Build workflow gains a Benchmark leg that runs make bench-ci and compares against tests/bench-baseline.json in report-only mode; bench-report.yml posts the result as a PR comment. scripts/ci/host-lock.sh keeps the other legs off the host while the Benchmark leg measures: they hold it shared for their own run, and the Benchmark leg holds it exclusive only around make bench-ci.

Changes from the last revision of #223, for the open review threads there:

  • The lock holder binds a control socket unique to each acquire and records its path in the pid file. Release reads that path, cleans up when the holder is gone, and keeps the lock state and exits 1 when a live holder does not answer, so a timeout no longer orphans a holder.
  • qemu and orbstack corpus prep run under their own BENCH_PREP_TIMEOUT (default 900 s) instead of the per-sample bound.
  • The result emitter iterates rows by metric and rejects a duplicate metric.
  • bench-timeit rejects an elapsed time under 1 us instead of printing 0.
  • The commentflow, format and pre-commit checks exclude tests/bench-corpus, and the corpus files have descriptive names.
  • The workflow keeps the existing comments on unrelated steps, and docs/testing.md only adds a benchmark section.
  • The fixture cache is written to a temporary copy and renamed into place.
  • When a later run is back inside the threshold, the earlier report comment is updated to say so.
  • An empty BENCH_LMBENCH_ENOUGH restores lmbench auto-calibration.
  • A failed lmbench download or build makes fetch-fixtures.sh skip lmbench instead of failing the whole fetch.

Publishing results to GitHub Pages is left for a follow-up PR.

Tested on current main (170b5df): make check passes, and make bench-ci passes with all three tiers producing every metric, in about 12.5 minutes. The four Tier 3 metrics have no baseline yet; a baseline from the CI host should be promoted with scripts/bench-promote.py after the first runs there.

Closes #195