fix(limits): isolate in-flight Redis I/O on a dedicated thread and partition fallback
Merged
RU
7h ago+781+781+012345678901234567890123456789-216-216-012345678901234567890123456789
Problem
- Main Tokio runtime starvation during request bursts:
ConnectionManager's socket pipeline task, claimEVALdeadlines, per-slot renewal tasks, release retries, and reconnect handshakes all ran on the serving Tokio runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large JSON request bodies occupy the worker threads with synchronous parsing long enough that Redis replies sit unread past the 250 ms claim deadline — even when Redis itself answers in tens of microseconds. Three such timeouts drop the healthy connection and trigger a TLS reconnect on the same congested runtime, causing multi-second fallback windows. - Unpartitioned fallback and split counters: When falling back locally, each replica admitted up to the full global
limitrather than its share across active replicas, did not count slots already admitted via Redis on that replica, and did not register fallback-admitted slots back into Redis once the connection recovered.
Change
- Dedicated
"inflight-redis"OS thread:SharedRedis::connectspawns a dedicated OS thread with its own single-threaded Tokio runtime that ownsConnectionManagerand executes all claims, 250 ms deadlines, withdrawals, release retries, reconnect loops, batched renewals, and pod heartbeats off the HTTP worker pool. Straggler timeouts/errors on an already-replaced connection no longer tear down the replacement connection. - Unified per-replica counter and scoped partitioned fallback: Every admitted slot on a replica (both Redis-admitted and fallback-admitted) is tracked in a single
PodInflightCounter. Each replica lazily heartbeats only the limiter scopes it actually serves ("realtime"vs"batch", 5 s interval / 15 s lease) so API replicas and batch daemon replicas do not inflate each other's replica counts, and local fallback caps concurrency atlimit.div_ceil(live_pods). - Batched renewals and reconnect re-registration: Replaces per-slot renewal tasks with a single 30 s batched
SYNC_KEY_SCRIPTpass (chunked at 512 members per key). Active fallback slots are immediately synced to Redis upon reconnect so global counts stay accurate across failovers and recoveries; fallback slots that finish before ever reaching Redis drop with zero Redis traffic.
Tests
claims_complete_when_caller_runtime_workers_block: freezes all threads of a 2-worker caller Tokio runtime for2 × REDIS_TIMEOUTwhile a claim is in flight and verifies the dedicated thread completes the claim with 0 timeouts and 0 withdrawals.fallback_splits_the_limit_across_live_pods_per_scope: verifies"realtime"and"batch"heartbeats track live replicas independently and fallback divideslimitacross live replicas in the same scope.fallback_counts_slots_already_admitted_through_redis: verifies slots admitted via Redis count toward the replica's partitioned fallback cap when Redis disconnects.fallback_slots_sync_to_redis_on_reconnect_and_release_on_drop: verifies slots admitted locally during an outage sync into Redis upon reconnect and sendZREMwhen dropped.- All 13 unit tests in
realtime_inflight::testsandcargo clippypass with zero warnings.