fix(broker): retry pressured control frames

Merged
KJ
15d ago
+2052-113

Closes #1915.

Outcome

  • Retries correlated agent.register d1_pressure on the same socket with the exact frame, bounded 1/2/4/8/8-second backoff and a 25-second wall-clock budget.
  • Surfaces persistent registration pressure as a named retryable spawn error.
  • Retries acknowledged agent.deregister, delivery.ack, and non-task action results without extending application liveness.
  • Retains pressured non-task action.result frames across exhaustion or disconnect and replays the same stable ID after reconnect; a replay that remains pressured waits for an organic reconnect instead of cycling the socket.
  • Returns task action.accept/action.result pressure immediately to the shorter-lived task owner.
  • Explicitly deregisters late successful registrations after either retry-count or wall-clock exhaustion; compensating cleanup is itself retryable.
  • Bounds one-way retry tracking and exhausted tombstones to one retry-budget window.
  • Maps only the exact proven pre-mutation registration failure shape to SDK/CLI spawn_retryable.

Verification

  • Real-socket transient and persistent tests cover agent.register, delivery.ack, action.result, and agent.deregister.
  • Separate agent.register regressions deterministically cover retry-count exhaustion and wall-clock-budget exhaustion. Both prove fail-closed caller errors and compensating deregistration of a late success.
  • Linux verification of the retry-count path passed 10/10 idle and 6/6 with the test pinned to two CPUs while 24 busy loops ran; the full node_control:: module passed 116/116 under load.
  • Linux mutation checks independently removed the fail-closed reply and compensating deregistration; each matching regression failed.
  • Disconnect-during-backoff proves exact non-task action.result replay on a second socket.
  • Regressions prove unpressured results are not replayed, repeated replay pressure does not force a reconnect loop, and task frames bypass the long socket retry budget.
  • Mutation proof: disabling each delivery.ack, action.result, and agent.deregister retry hook made its matching regression fail.
  • cargo fmt --all -- --check and git diff --check pass.
  • Exact Rust and TypeScript gates run in GitHub CI. Main has the known #1909 shutdown_does_not_block_on_a_full_fleet_queue flake.

[!NOTE]
Medium Risk
Touches fleet registration, deregistration, delivery acks, and spawn error mapping under load; bounded retries and compensating cleanup limit blast radius but wrong retry semantics could duplicate or leak agents.

Overview
Adds bounded d1_pressure recovery on the fleet node-control WebSocket so transient Relaycast pressure does not drop registrations or one-way control traffic.

Broker (node_control) introduces a shared retry policy (exponential 1/2/4/8/8s backoff, 5 attempts / 25s wall clock, under the fleet registration waiter). agent.register retries the exact frame on the live socket; persistent pressure fails the caller with a named d1_pressure error, and late successes after exhaustion trigger compensating agent.deregister (also retryable). agent.deregister, delivery.ack, and non-task action.result are tracked and replayed on the same socket; exhausted or disconnected non-task results are retained and replayed after reconnect with stable IDs. Task accept/result pressure still surfaces immediately to the shorter-lived task owner.

SDK/CLI adds spawn_retryable when spawn confirmation sees the proven pre-mutation agent.register + d1_pressure failure shape; other d1_pressure strings in errors stay spawn_failed.

CHANGELOG documents the fleet-spawn and control-frame behavior.

Reviewed by Cursor Bugbot for commit 54c441210ca3cdd07ac66530819bec300902f077. Bugbot is set up for automated code reviews on this repo. Configure here.