fix(trpc): move a boot to a new lock id before its renewal check

SU
1d ago
+9-3

Problem

When Autumn has a short outage, a cloud-workspace boot can get stuck in its settle loop. Every settle of that boot fails with 409 lock_already_exists, every 30 minutes plus QStash retries, until the box stops. While it is stuck, the boot never renews, so its spend cap is not checked. In Sentry (API-3W and API-3X, about 150 events in total), each 409 run starts right after a burst of Autumn 503s (10-28 06:00 UTC and 10-30 06:00 UTC). One run went on for about 17 hours.

Root cause

settleBoot renews by calling autumnClient.check with lock id <bootId>:<lockSeq+1>, and it saved lockSeq only after the check returned. If Autumn creates the lock but the response fails (the 503s), lockSeq stays where it was. Every retry then asks for the same lock id, and Autumn answers a repeated lock id with 409 lock_already_exists.

Fix

Save lockSeq + 1 in the same write that records the billed window (finalizedThrough, lockOpen: false), before the renewal check. A retry after a failed check then asks for a fresh lock id and gets Autumn's real allowed/refused answer. A lock the failed attempt may have created is left to expire, and the SDK documents on Lock.expiresAt that expiry "releases the held balance". authorizeBoot's fail-open path already accepts the same kind of orphaned lock.

The first revision of this PR treated the 409 as a granted renewal. Review rejected that, because it assumed a refused check never records its lock id, which nobody has verified. This revision removes it and makes no assumption about Autumn's refusal behaviour.

Verification

$ bunx biome check packages/trpc/src/router/cloud-workspace/meter.ts packages/trpc/src/router/cloud-workspace/meter.test.ts
Checked 2 files in 44ms. No fixes applied.

$ cd packages/trpc && bun run typecheck
$ tsc --noEmit --emitDeclarationOnly false
(no errors)

# before the fix, with the lockSeq assertion added to the failed-renewal test
$ bun test src/router/cloud-workspace/meter.test.ts
(fail) settleBoot > a retry after a failed renewal asks for a new lock ...
 26 pass
 1 fail

# after the fix
$ bun test src/router/cloud-workspace/
 44 pass
 0 fail
Ran 44 tests across 3 files.

Not in scope

This fixes only the renewal 409 loop. An earlier triage of this issue found a second path from the same outage that this PR does not change: if a finalize confirm returns 503 after Autumn applied it, the retried finalize gets 400 "lock not found". recordUsage reports that and falls back to track, which can bill that window twice. Fixing that needs an answer from Autumn on what a 503 means for finalize and whether Idempotency-Key is honoured there.

Review notes

  • Antagonist review: the first revision (treating the 409 as a granted renewal) got DO NOT MERGE, because it rested on an unverified claim about Autumn. This revision got MERGE.
  • The Greptile finding about JSON spacing in the 409 matcher refers to code from the first commit, which this revision removed. It does not apply to the current diff.

This is billing code, so a human must review and merge it.

Refs API-3W