feat(backend): durable first-refusal clock for producer-pending seals

Merged
GI
3h ago
+310-7

Problem

Producer-pending seal refusal retained retry state only in pusher memory. A process death or legacy retry exhaustion lost the first-refusal age even though custody tokens remained durable.

What changes

Persist record_seal_refused_at once inside the existing refusal transaction, preserve it on retries, and clear it atomically on successful seal_record_audio closure. Emit bounded job-page refused count/oldest-age observations in the sweeper summary, declare the state/writer, and add strict transaction and emulator pod-death regressions. Include the changes document and all three flag registry files.

Flag: RECORD_SEAL_REFUSED_STAMP_ENABLED, default true, approved by lead2 (bus 430238). False restores zero-write refusal and disables sweeper observations. Successful seal_record_audio closures still clear prior stamps.

Rollback: set the flag false on relevant processes, or revert the PR. Old stamps can remain ignored; no backfill or cleanup is needed.

Custody/privacy: one content-free epoch field, bounded to the first refusal update inside the existing transaction. No audio or identifiers added; no token, seal-authority, privacy-proof, retry-budget or is_debt changes. No .github changes, GCP tier, production access, merge or deploy.

Validation

  • Targeted backend suites: 1,093 passed, 75 strict expected failures (state-table obligations); includes record seals, sweeper/rules, state-table/writers, worker/sweeper deployment, rendered deployment contracts, simulator helpers.
  • Lifecycle emulator simulator: 5 passed, 3 existing strict expected failures. New regression makes the refusal in the pusher child, SIGKILLs it, retries in a replacement child, observes age/count, and verifies exact-token drain plus atomic seal clearing.
  • Simulator ran with a private child home/config and a temporary runner wrapper allocating a free emulator websocket port; the stock runner first rejected ambient CLOUDSDK_PYTHON and reachable host ADC, then hit a shared-port collision. No credential preflight or assertion was bypassed; all owned children were shut down.
  • Feature flag registry checker passed (pre-existing advisory warnings). Pinned formatting passed.
  • First make preflight ran its checks but rejected tracked-file edits made concurrently during test refinement. Frozen-tree make preflight passed all 43 checks (351.45s); scripts/pr-preflight --pr-body-file passed all 43 checks (260.36s), and its metadata-only pass also passed. First normal pre-push passed all 43 manifest checks and its Firestore guard (149.81s), then caught an inferred dict[str, bool] type for the nullable clear. Added an explicit dict[str, bool | None] annotation; the second normal pre-push passed, including all 43 manifest checks, zero-error typecheck, seven selected backend test files and pinned formatting. Its Firestore guard passed in 199.34s with no timeout. No bypass, force push or custom cap was used.
  • Original required Grok review: exec-address-clay-code-purchase-bonus-index, resolved by agentctl to Grok on Cursor. Accepted narrow monitoring scope; flagged proof-repair seal clearing outside the named files. Harness import placement corrected. Parent independently verified the finding.

Read-side scope ruling

The refusal clock is meaningful only while record_audio_closed is false. Closed stamped rows are stale metadata and are excluded before the sweeper updates backlog, refusal count or maximum refusal age. Regressions cover a closed stamp alone, an older closed stamp beside a newer open one, and a real proof-repair seal that retains the stamp but is ignored on the next sweep.

Per uicleanup's chat decision, repair/nojob writers and seal authority remain untouched. The normal successful seal_record_audio branch still clears the timestamp in its existing transaction. The state table and changes doc now specify the unclosed-only read contract. The previous scope ask is withdrawn and no longer blocks readiness. A reopened row may still carry its original first-refusal epoch; this PR does not redefine that clock or change a writer to reset it.

The required Grok review was completed on the original implementation; this follow-up implements the lead's read-side resolution of its retained-clock finding. No further scope extension is requested.

Read-side follow-up validation at add5c6e1254cca208c32040f38792baa07f8315b:

  • Targeted sweeper/budget, lifecycle state/writer inventory and normal seal suites: 1,020 passed; 75 existing strict expected failures. All three new closed-stamp/repair-sealed cases passed.
  • make preflight: all 43 checks passed (160.61s). PR metadata validation passed.
  • Normal pre-push passed, including zero-error backend typecheck, selected backend tests and formatting. No bypass, force or custom cap.
  • Repair/nojob writers, seal authority and the normal successful-seal clear are unchanged by this follow-up. CI is tracked on this exact PR head.

Measurement limits

No production stock/age, deployed flags, legacy-drain prevalence, released-client field consumption or end-to-end production recovery was measured. Sweeper observations cover only its existing bounded record_pending job page, before apply, and do not cover no-job rows. This PR emits structured summary samples, not a deployed external metric/alert. Strict expected failures preserve missing authoritative fleet gauge/committed alert and automatic requeue/deadline obligations; phase derivation stays unchanged.

Failure-Class: none
Failure-Class-Reason: Phase 1 adds a durable age observation; it preserves existing retry and custody authority. Behavioral guards cover timestamp idempotence, atomic clearing, legacy exhaustion and process death.