feat(runner-watch): structured gate, value normalizers, coverage map, tree-sitter deltas

Merged
BN
1mo ago
+1935-223

What

Hardens runner-watch against four gaps where conformance could silently degrade or under-detect upstream drift.

#1 — Structured verdict, not markdown-grep

The gate no longer scrapes the human markdown report. compare::analyze() returns a structured ConformanceReport; conform() gates on report.failures(&GatePolicy) and writes a machine-readable conformance-result.json (plus per-scenario JSON). render_report is now a human-only artifact. Removed the four string-scraping fns (status_mismatch_in_report, bracketed_statuses, schema_mismatch_in_report, schema_diff_removes_official_fields) and their fixture tests. Gate semantics preserved: status-ignore for /oauth2/token+/messages, request-schema-any-diff, and acquirejob response-field-drop via a structural schema_drops_fields (removals/type-changes fail, additions tolerated).

#4 — Declarative value normalizers + optional value gating

normalize_value() recursively collapses volatile scalars (GUID/JWT/gh-token/URL/timestamp/big-int/high-entropy) while keeping stable values visible, so real value regressions remain detectable. New ValueGate policy; default Off preserves legacy pass/fail and only reports normalized value diffs. --value-gate-strict opts into gating normalized response values for all shared endpoints except known-volatile ones (OIDC, tokens, connectionData, signed blob URLs). Strict is opt-in on purpose: validating its false-positive rate requires a live golden replay.

#2 — Endpoint coverage map

New coverage.rs + runner-watch coverage --runner <v> [--strict]. Cross-references golden-exercised endpoints against routes scraped from crates/preloop-runner-server/src/routes.rs, restricted to the runner-facing protocol surface. Warn by default; --strict fails on uncovered runner-facing routes minus .runner-watch/coverage-allow.txt. Against the committed v2.336.0 corpus: 18 covered, 103 uncovered alias routes, and 3 genuine gaps — the runner calls /_apis/distributedtask/pools, /_apis/distributedtask/pools/{p}/agents, and /_apis/oauth2/token/{p}, which the server serves only under /runner/server/… or /_apis/v1/….

#3 — tree-sitter-c-sharp delta extraction

Replaces line/substring C# parsing with tree-sitter-c-sharp, detecting fields, record parameters, enum members, methods, and expression-bodied properties (recall win), with a heuristic fallback. DeltaEntry/change_type contract unchanged.

Verification

  • cargo test -p runner-watch: 47 passing, incl. gate-the-gate behavioral tests (status mismatch, acquirejob field-drop, value-gate ignores-volatile-but-catches-regressions, missing-endpoint) and coverage tests over real golden fixtures.
  • cargo fmt --all --check; just clippy (workspace, -D warnings).
  • runner-watch coverage --runner 2.336.0 exercised against the real corpus (numbers above).

Not covered here

  • just conform (live server + mitm/GitHub) and zizmor (pre-existing failure in .github/workflows/release.yml) were not run locally; prod CI covers them.

Protocol surface

No runner-facing wire changes — tooling only (crates/runner-watch). Deps added: tree-sitter, tree-sitter-c-sharp.

Summary by cubic

Replaces the markdown-scraping conformance gate with a structured ConformanceReport, adds an optional strict value gate and a coverage subcommand, and switches C# delta extraction to tree-sitter-c-sharp. Job-level env: is now serialized as TemplateToken maps the official runner accepts, fixing jobs that previously failed with "The template is not valid".

Gate changes

  • The verdict comes from compare::analyze() instead of scraping the markdown report; it writes conformance-result.json and per-scenario JSON.
  • normalize_value() collapses volatile scalars (tokens, GUIDs, timestamps, large numbers) while keeping stable values visible, and compares every response occurrence aligned by flow slot — null/absent bodies are treated as absent, and only the shared occurrence count is compared.
  • --value-gate-strict gates normalized response values and is off by default, preserving legacy pass/fail behavior.

Coverage and parsing

  • Adds runner-watch coverage to cross-reference golden endpoints against server routes; against the v2.336.0 corpus: 22 covered, 67 uncovered, and 1 genuine gap. It is method-aware, matches catch-all routes by prefix, canonicalizes transport aliases, and --strict fails on uncovered runner-facing routes minus the allowlist.
  • C# delta extraction now uses tree-sitter-c-sharp with method signatures that encode generic arity and parameter passing modes; the line heuristic remains a fallback when parsing fails.
  • Job-level env: values serialize through template_string_token, so runtime-only expressions (${{ github.workspace }}/bin) reach the runner as expression tokens rather than raw literals.

Summary by CodeRabbit

  • New Features
    • Added coverage reporting with HTTP method awareness, catch-all route matching, endpoint alias normalization, and allowlist filtering.
    • Added optional strict response-value validation and structured JSON comparison artifacts.
    • Improved C# route extraction, including support for overloaded methods.
  • Bug Fixes
    • Comparison now detects schema and value differences across matching request and response occurrences.
    • Improved handling of large numeric response values and runtime expressions in job environment variables.
  • Tests
    • Expanded coverage, extraction, replay, environment propagation, and comparison validation.