[eval][cli] fix: harden source digest, rubric parsing, and secret redaction

JO
3d ago
+84-30

What

  • source_digest (osmosis_ai/source_scan.py) length-prefixes extra, each relative path, and each file's contents before hashing, so different source trees can no longer hash the same byte stream (previously {"ab": b"c"} and {"a": b"bc"} produced the same digest).
  • extract_assistant_content (osmosis_ai/eval/rubric/dataset.py) skips non-dict entries in messages, so a row like {"messages": [1]} reports the usual line-numbered CLIError instead of raising AttributeError with a traceback.
  • Secret redaction (osmosis_ai/platform/cli/secret_redact.py) routes string values, dict keys, and redact_secret_text through one helper that replaces the whole string with [REDACTED] when the marker would join its neighbours into the secret again (for example "]]]" with the secret "]]" previously became "[REDACTED]]"), and with an empty string when the secret is part of the marker itself (for example "RE"). redact_provided_secrets now replaces all secrets in each string and checks the result against all of them, so one secret's marker cannot rebuild another ("yx" with secrets "]x" and "y" previously became "[REDACTED]x").
  • One regression test per fix.

Why

These came out of writing "fuzz test as spec" style properties for pure-Python trust-boundary functions: the digest should be injective over source trees, rubric loading should fail only with CLIError, and redacted output should never contain the secret. Each property was violated by a small, concrete input.

The digest collision matters most: the same digest keys the Harbor bundle wheel cache, pins the resume lock of named local eval runs, and is recorded as source_digest in uploaded eval manifests, so a collision could reuse a stale bundle or let a run resume across a code change.

Compatibility: digest values change for every project. The Harbor bundle cache misses once and rebuilds. A named local eval run created by an earlier SDK refuses to resume after upgrading, reporting that the rollout source changed. Newly uploaded manifests carry the new digest values. No public API changes; I did not mark this [BREAKING], but the release notes should mention the resume impact.

How to Test

  • uv run pytest tests/unit/eval/local tests/unit/test_packaging.py tests/unit/test_rubric_cli_command.py tests/unit/platform/cli/test_secret_redact.py tests/unit/rollout/test_harbor_backend.py → 754 passed.
  • After the marker follow-up: uv run pytest tests/unit/platform/cli tests/unit/cli → 1528 passed.
  • uv run ruff check . and uv run ruff format --check . → pass.
  • uv run ty check --error-on-warning osmosis_ai/ → pass.
  • git diff --check origin/main...HEAD → clean.
  • Full pytest -n 4 --dist worksteal locally: 3733 passed, 11 failed, all in tests/unit/test_dependency_security.py::test_wheel_verifier_checks_allowed_versions. My local environment has stale installed osmosis-ai metadata from before #457 (uv<0.13,>=0.9.7), which the wheel verifier reads; unrelated to this diff, CI builds fresh metadata. test_a_second_interrupt_during_server_shutdown_cuts_its_grace_short failed once under xdist and passed on rerun.
  • Not run locally: uv run --no-editable pyright --verifytypes osmosis_ai --ignoreexternal (no public API changes); left to the typecheck-pyright CI job.

Checklist

  • PR title follows [module] type: description format
    (labels are derived from it automatically — no need to add them by hand)
  • ruff check . and ruff format --check . pass
  • ty check --error-on-warning osmosis_ai/ passes; uv run --no-editable pyright --verifytypes osmosis_ai --ignoreexternal has no unexpected errors (see CONTRIBUTING.md for the Harbor baseline)
  • pytest passes (new tests added if applicable)
  • Public API changes are documented
  • No secrets or credentials included