Replace #132: pack fixtures, fix CI, remove hash pinning

Merged
SU
13d ago
+1442405-22801

Replaces #132. It carries the same fixed-income workflow work as #132, rebuilt on main so that 5.9 GB of JSON fixtures never enter Git history. It also fixes the problems that kept #132's CI red, and removes the hash pinning that made every edit to the repository need a matching hash refresh.

What was wrong with #132

  • 5.9 GB of fixtures in Git. #132 committed 631 regression corpora of 1 MB or more. Seven of them (979 MB) were in Git LFS; the rest (5.2 GB) were plain blobs. Every CI checkout pulled all of it, and LFS bandwidth was spent on every job.
  • Test and Bounded Rust Assurance ran out of runner disk. About 70 include_str! sites embedded 60–180 MB corpora into hundreds of test and example binaries, on top of the checkout. Both jobs died with No space left on device. Most of the 935 assurance steps never ran.
  • Excel Add-in (Windows) had two latent failures.
    • The library parity input curve.fit.dated came from exp_m1. MSVC's libm returns a result one ulp away from glibc's, so the emitted request no longer matched cases.json.
    • The parity steps passed --binary target/excel/examples/X.exe to Python subprocess. Windows CreateProcess rejects forward-slash relative paths. The job never got this far before.
  • The release would have failed to publish. convex-analytics now depends on the new convex-rates and convex-credit crates, but release.yml still published only core, math, curves, bonds and analytics.
  • Every edit needed hash bookkeeping. The assurance tooling pinned the SHA-256 of nearly every protected file in four places (numeric-evidence.json, inventory.json, library-inventory.json, the comparison catalog) and required a change-reviews.json row with fresh hashes for each change. Any fix, even a clippy one, failed CI until those were refreshed.

Commits

  1. Bring over the #132 workflow tree without its large fixtures. #132's final tree (feb8e889), unchanged except that the 631 large fixtures are left out.
  2. Pack the large fixtures and fix the CI and release failures.
    • Each fixture of 1 MB or more is committed as <name>.json.xz: 14.2 MB in total, down from 5.91 GB. Git LFS is no longer used.
    • python3 tools/fixtures.py materialize writes the JSON beside each archive (about 80 s); the written files are Git-ignored. pack refreshes an archive after a deliberate regeneration.
    • The 67 include_str! sites that pointed at packed corpora, in 56 test and contract files, now call packed::text(...) from tests/fixtures/packed.rs. Each file is read once per process. Nothing else in those files changes; I verified this token by token.
    • extended.rs uses the two quote literals already in cases.json instead of recomputing them with exp_m1. The Excel job passes Windows-style binary paths.
    • Test, Bounded Rust Assurance, Excel and Java materialize fixtures instead of fetching LFS. The two Linux heavy jobs free unused preinstalled toolchains and build with CARGO_PROFILE_DEV_DEBUG=0: with line tables only, the workspace test build passed 32 GB before running a single test.
    • convex-rates and convex-credit are published before convex-analytics.
    • docs/TEST-DATA.md and README.md describe materialize instead of git lfs pull.
  3. Remove hash pinning and change-review records from assurance.
    • Deletes tools/assurance/change-reviews.json, numeric-evidence.json and tools/source_pins.py, and the checks that read them.
    • Drops sha256 from the inventory.json rows and from the comparison catalog and pack references. library-inventory.json lists inspected_sources as paths; its observation anchors are still checked against the current files.
    • Removes tests/fixtures/packed.json. The archives are the source of truth, and xz's CRC64 rejects a corrupt one. materialize and check also require tests/fixtures/.gitignore to match the archives.
    • Kept:
      • The structural checks: forbidden dependencies, lints, lifecycle purity, required CI steps, inventory paths and exception expiry.
      • The solver-default check (DEFAULT_TOLERANCE 1e-10, DEFAULT_MAX_ITERATIONS 100) and the required comparator and reference files.
      • The --base checks that reject wider tolerances and removed cases, measures or library coverage.
      • The hashes recorded inside fixture data and capture manifests, and the QuantLib wheel download checksum.
  4. Fix the offline comparison and Excel jobs found on the first full CI run.
    • Offline Quantitative Comparison now materializes the packed fixtures. It also frees runner disk and drops debuginfo, like the other jobs that hold the unpacked corpora.
    • The Excel tests checked CX.LIBRARY.RANGE output by rejoining it with LibraryCells.Join, which applies the 2 MiB request bound. Fourteen P&L responses are larger; the native response bound is 16 MiB. The tests now rejoin output with a test helper, ResponseCells.Join, that checks cell sizes only. The request-side round trips are unchanged.
    • LibraryCells.ValidateInput let EncoderFallbackException escape for an unpaired surrogate. It now throws the plain ArgumentException its test expects.
  5. Check the optional graph hedge controls within libm drift. Bounded Rust Assurance failed at step 600 of 945, which #132's CI never reached.
    • The optional graph hedge reference, qualification and calibrated generators call the platform libm (expm1, erfc, exp). Their regenerated floats differ in the last digits on Linux and on Windows, and both differ from Codex's committed controls, so their byte-exact checks can pass only on the machine that wrote them.
    • Regenerating would cascade: the controls' bytes, and the reference script's own bytes, are recorded as source reads in other fixtures.
    • So the generators and controls are unchanged, and the three CI steps (same names) now run tests/scripts/optional_graph_hedge_controls.py. It regenerates the same six files, requires structure, text (including source reads) and integers to match exactly, and allows floats 1e-11 absolute, or 1e-13 relative above magnitude 100. The largest drift is 1.4e-12 on Linux and 2.8e-12 on Windows; the tightest consumer tolerance in the controls is 1e-10 (cash).
    • The generators' own check modes stay byte-exact, so they still pass only on the platform that wrote the controls.

No financial code, fixture value, expected result, tolerance or solver default changes. The only assertion changes are the Excel output round trips in commit 4 and the three optional-graph-hedge control checks in commit 5. #132's CI never reached dotnet test, so this PR is the first run of those tests.

Verification

Linux (Ubuntu under WSL 2):

  • The full workspace suite (cargo test --locked --all-features --workspace --no-fail-fast -- --test-threads=1, CARGO_PROFILE_DEV_DEBUG=0) gave 3,351 passed, 0 failed, 78 ignored. It ran before the clippy fixes in packed.rs and the contract files' module lines. After those fixes, I re-ran cargo fmt --check, cargo clippy --locked --workspace --all-targets --all-features -- -D warnings (0 warnings) and the fixture-reading tests (monotone_contracts, child_maturity, concurrent_claims, custody_lineage, date_rolls).

  • A regeneration probe found that 344 generator outputs reproduce the fixture bytes exactly.

  • At the final head:

    • tools/fixtures.py check and reconciliation/comparison.py check passed.
    • tools/assurance/check.py passed both with and without --base acc357b8.
    • The unit tests passed: assurance 37, library 64, comparison 26 and swap QuantLib 18.

  • Commit 5: the new checker passes on Linux and Windows. It rejects a 1e-10 change to a price, an edited source-read hash, a changed tolerance, a dropped case and a float turned into an integer. check.py with and without --base, the assurance tests and tools/library/check.py passed. The other 154 Python steps after step 600 that need no build pass on Linux. The 51 Python steps that need a built binary and the 134 cargo steps after it were first run in CI.

  • Commit 4: in a fresh clone with no materialized fixtures, the full Offline Quantitative Comparison job passed: 392 retained QuantLib rows, 973 swap values, 35,820 CDS values and the credit calibration. check.py and the assurance tests passed again with the new workflow.

  • CI on b111510e (run 39447079562): every job passed, including all 946 steps of Bounded Rust Assurance, Excel Add-in (Windows) with dotnet test, Test, Offline Quantitative Comparison, and the Java and QuantLib workflows.

Windows:

  • The Excel job's library_workflow parity (110 cases) and the five parity steps after it passed.
  • The Excel test project builds, and the managed RawJsonChunks… test passes.

Not verified here

  • The full Excel job (36 release builds plus dotnet test) was replayed locally only through its first six parity steps. CI is its first full run, including the 14 P&L P/Invoke tests fixed in commit 4.
  • This PR does not review #132's 203k lines of Rust and 39k lines of Python in depth. That work was produced and self-attested by Codex. CodeRabbit skipped #132 because it has too many files.
  • With byte pins gone, changes to fixtures and comparators rely on pull-request review. The --base checks still block wider tolerances and removed coverage.
  • As in #132, no library, model, platform or production certification is claimed.

🤖 Generated with Claude Code