harden(pusher): persist playback projection rebuild debt

Merged
GI
5h ago
+1496-73

Problem

Pusher drops its in-memory playback projection close obligation when the bounded close rebuild retries are exhausted or the socket is evicted. The retained audio remains safe, but no durable owner retries the stale playback metadata after the socket disappears.

What changes

  • Persist coalesced record_projection_rebuild_due markers in record_projection_rebuild_jobs, keyed by user/conversation, on exhausted retries and socket cleanup. No conversation marker and no additional ordinary-flush reads or writes.
  • Queue terminal handoffs in one bounded metadata-only background task. Preserve a separate bounded overflow retry inventory, batch writes with one-second RPC deadlines, and cap all socket-cleanup marker work at three seconds. Marker failures never change recording completion.
  • Retire confirmed empty-inventory markers after seven days from Firestore creation, or under the 10,000-marker maintenance ceiling. Limit collection scans to 10,001 and rebuilds to 128; expose bounded count, observed oldest age, capacity state and retired_no_objects. Retirement rereads the marker version and the conversation producer map in one Firestore transaction. Only an explicit empty map permits deletion; live, unknown or malformed producer fences retain and rotate debt, including under collection pressure. The pressure-blocked gauge and aggregate fenced count surface passes that cannot evict empty markers. No audio is deleted.
  • Extract shared full metadata discovery and a socket-independent rebuild. A bounded pass in the existing record-seal-repair CronJob consumes at most 128 markers in 60 seconds, rotates work before discovery, and clears only the version it attempted. Failed, interrupted, invalid, or concurrently replaced work remains due; confirmed empty inventory follows the age/capacity policy above. A rebuild may update metadata while existing producers remain pending, but keeps its marker until those fences clear through their existing owner.
  • Refuse deleted owners before discovery and recheck deletion intent, privacy, and the prior audio-files revision transactionally before updating only audio_files. Keep preview read-only and retain debt on a broken raw-pack index.

The runtime kill-switch parser lives in config/batch_pipeline_projection_rebuild.py; metadata rebuild stages do not read process policy from the environment.

Flag: PUSHER_PROJECTION_DEBT_DURABLE_ENABLED, default ON. 0, false, or off disables marker handoff and consumption. Rollback: set recordSealRepair.projectionDebtDurableEnabled=false in the repair runtime first, then disable pusher's flag; existing markers remain inert and can resume when reenabled. This PR does not unsuspend or activate a CronJob.

Custody and privacy

Playback metadata only. No seal authority, producer-token mutation, heartbeat, entry clock, dead-letter, conversation status, PCM read, or audio-object creation/rewriting/deletion. Tombstones, hard/soft deletion, account deletion, privacy changes, sync opt-out, peer projection updates, and marker replacement are fenced. Scope is redesign phase 5C only. #24712/#24716 and #23850 were checked before editing; #24712 and #24716 merged during verification and their flag metadata is preserved in the feature branch; shared overlap is flag metadata, not projection-drain behavior or deployment configuration. No .github changes.

A persistent Firestore outage or process death before a marker is committed can still prevent handoff; this marker is not a WAL. Failed marker writes retain metadata-only retry inventory and emit sanitized marker_write_failed outcomes; bounded inventory saturation emits marker_retry_overflow. Cached close debt remains retryable even when overflow inventory is full.

Verification

  • All tests/build checks ran through ~/workspace/omi/bin/omi-heavy with the requested GCP credential/project variables unset.
  • Focused per-file backend runner: 314 passed (projection producer/consumer, flag budget, rendered independent rollback, CLI output isolation, batch-stage environment ratchet and outbox fault injection).
  • State table/invariants/writer inventory plus other deployment contracts: 1,917 passed, 75 baseline xfailed.
  • Lifecycle simulator plus lifecycle properties under the existing isolated stack environment and a disposable loopback Firestore emulator: 23 passed, 3 existing strict xfailed. The shared-collection fixture includes a clear peer and verifies per-owner transactional outcomes. The exhaustion/eviction/deletion/marker-race cases and six empty-retirement cases passed: aged live/clear fences, pressure with fenced markers, post-discovery producer registration, marker replacement and late-tail repair.
  • One local composed full-stack attempt stopped at an HTTP read timeout in the existing REST-restart scenario before reaching lifecycle tests; CI passed that scenario on the same production source. The final exact-head CI gauntlet is linked below.
  • Runtime image dependency closure: 19 registered images passed; no image/dependency changes needed.
  • Required Grok review via agentctl: Luna R1 and R2 findings fixed. The same required Grok reviewer rechecked transactional empty retirement, producer and marker races, pressure telemetry and regression coverage: no blockers. No additional reviewer agents.
  • make preflight: 44 checks passed, including the registered Firestore guard. scripts/pr-preflight metadata contracts passed with the documented six-line router exception; the final frozen run passed all 44 checks; the normal hook repeated them successfully on the published head.
  • Normal pre-push passed on 95a9ea47ae98eb03843909459a6604523534a98e: full backend type checking, selected backend units, registered Firestore guards, OpenAPI and pinned formatting. No bypass, cap override or force push; Firestore guard passed in 172.30 seconds without timeout or retry.

Exact published head: 95a9ea47ae98eb03843909459a6604523534a98e. CI results: checks for this exact commit.

Product invariants affected

none

Not measured

No GCP/prod access or tier used; no live IAM/permission test, deployment, CronJob activation, production backlog/cost/latency measurement, device playback acceptance, or A/B worker-death/dead-letter simulator changes. Existing strict xfails remain existing simulator/state-table debt.

Line-Count-Exception: backend/routers/pusher.py | 1851 -> 1857 | Phase 5C adds exact-owner terminal projection-debt and socket-cleanup handoffs at the existing boundaries; full discovery is extracted into a shared module, and splitting adjacent custody code would exceed this lane's named scope.