feat(batch): add opt-in dedicated projection drain pump

Merged
GI
12h ago
+613-22

Problem

Projection sweep supply is tied to serving instance count. Moving discovery onto dedicated instances needs an independent loop that cannot compete for maintenance/recovery leadership. This implements David's 2026-11-05 pump plan, sections 2 and 4; deployment shape remains pending.

Fix

  • Add serving_projection_sweep_enabled() using the shared kill-switch helper. BATCH_PIPELINE_SERVING_PROJECTION_SWEEP is on unless explicitly off; it gates only Serving.reconcile()'s sweep, preserving wakes, held drains, metrics and maintenance/recovery.
  • BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=true selects one lean projection loop instead of the three serving loops, independently of BATCH_PIPELINE_SERVING_ENABLED. It never invokes reconcile, maintenance or recovery dispatch. Sweeps use run_blocking(sync_executor, ...); owed drains start on the event loop. After each sweep, it schedules discovered or owed projection follow-ups via start_followups(()) and the owed index flush on the event loop, before starting owner drains after all pages. Each completed tick, including a failed tick, updates omi_batch_projection_pump_last_tick_seconds (Unix timestamp, zero before first tick). It logs successful sweeps, owners requested and elapsed time; errors log and retry.
  • Pump metrics scrapes use refresh-only regardless of the serving/read-only metrics controls; they never invoke the reconcile coordinator on the configured pump role.
  • Bound pages/tick (default 1, range 1-4), sleep (default 20s, range 2-60s) and owner drains (default 8, range 1-16); malformed values retain defaults. Register all five env names, and regenerate the registry.

All unset defaults preserve today's behavior. projection_drain.py changes only its capacity constant and the config import. No deployment files or .github/** changes.

Validation

Code and hermetic tests only; no production access or GCP tier used.

Merged origin/main at 148bf4c1c57073dba4ec9a750415f2c72f5adb87 using a merge commit, without rebase. Six files had conflicts: batch config, flag budget, kill-switch helper, pipeline contract docs, policy tests and projection drain. Both #24710 claim-aware behavior and pump capacity accessor are retained. The drain implementation is byte-for-byte main after removing only the capacity accessor import and restoring the old constant. Main advanced during CI; a second merge includes f8edd96cdbe (#24716). Its four conflicts in flag budget, kill-switch helper, rendered registry and policy tests retain the new striping controls alongside the pump controls.

bin/omi-heavy run -- env BACKEND_UNIT_TEST_FILE_LIST=<file> bash backend/test.sh:

  • Pump: 92 passed.
  • Serving (including four pump metrics-route combinations): 61 passed.
  • Projection follow-ups: 36 passed.
  • Owner drain / claim-aware admission: 66 passed.
  • Metrics: 32 passed.
  • GMP sidecar: 117 passed.
  • Policy: 76 passed.
  • Projection simulation: 1 passed, 2 slow tests deselected by the standard runner filter.
  • Stage env ratchet: 7 passed.
  • Flag budget: 24 passed.

Additional post-merge suites: projection discovery 67, follow-up desync 36, follow-up reads 6, sweep owner chain 34, and timeliness 18 passed.

Total: 673 passed, fifteen isolated files (plus the same 2 slow simulations deselected). Registry validation passed; the env-ratchet baseline is unchanged. The enforced push gate passed 41 checks; pyright reported 0 errors; selected push-time unit tests passed.

The requested merge reconciliation and metrics-route P1 are included. The listed P2 follow-ups are outside this change; no new review round was started.

Risk

Enabling a pump adds discovery work and shared-executor load. Bounds are pages 1-4, tick 2-60 seconds and owner drains 1-16; defaults remain 1, 20 and 8. The pump stays disabled by default. Durable receipts, owner claims and fencing are unchanged.

Ramp plan

Owner: lead2, after David's deployment-shape decision. Studio safe start on a dedicated service:

  • 4 instances.
  • BATCH_PIPELINE_PUMP_SWEEP_PAGES=1.
  • BATCH_PIPELINE_PUMP_TICK_SECONDS=2.
  • BATCH_PIPELINE_MAX_OWNER_DRAINS=8 per instance.

Never set BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=true on backend-integration itself: the toggle is service-wide. Keep BATCH_PIPELINE_SERVING_ENABLED false/unset on the dedicated pump service.

Cutover gate: owner_drain advanced >=20 rows/s for 30 minutes before disabling serving sweeps. Install and verify a missing/stale alert on the dedicated service's omi_batch_projection_pump_last_tick_seconds beforehand, including never-observed zero and missing instance series; a stale/missing alert must prevent cutover. Fallback: serving sweeps stay ON until both the throughput gate and heartbeat-alert prerequisite pass.

Alarm if Firestore read rate rises more than 50% over pre-cutover. Retain the owner_drain rows/s floor of 70% of pre-cutover; watch sweep elapsed time and executor saturation. Evaluate claim_busy share (no higher than pre-cutover) after serving sweeps are switched off, not during dual-running. This PR authorizes no deployment or cutover.

ROLLBACK

Env only, through new revisions, in this order:

  1. Restore serving sweeps ON on the existing service first (BATCH_PIPELINE_SERVING_PROJECTION_SWEEP=on).
  2. Set BATCH_PIPELINE_PROJECTION_PUMP_ENABLED=false on the dedicated service.
  3. Restore BATCH_PIPELINE_MAX_OWNER_DRAINS=8.

Failure-Class: none

These are pre-merge qualification corrections to the newly introduced pump role; no existing semantic failure-class definition matches the missing follow-up scheduling, pump-specific capacity bounds, and metrics-route role isolation.