Let a redis lock's waiters hear of its release before the releaser can retake it

Draft
MA
7h ago
+97-11

Found in the reflex 0.11.0a1 pre-release testing (FINDING-030 side note / FINDING-033 investigation; artifacts on branch claude/vibrant-pasteur-3921g3, prerelease-testing/runs/2026-11-11-v0.11.0a1/redis_runtime/ and verify_rds/). Independent of #7795 (reproduces with oplock off); touches the same file in different hunks and merges cleanly with it.

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Changes To Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Defect

With the Redis state manager, an instance that modifies a token back to back (e.g. a loop of app.modify_state calls) could repeatedly re-take the token's lock ahead of a waiter on another instance. Bisected to #7534 (d36a68707): the lock release changed from GETDEL to the compare-and-delete EVAL script (_RELEASE_LOCK_SCRIPT). Redis flushes queued replies in the order each client's first reply was queued: with GETDEL the keyspace notification to the waiters' subscriber is written before the releaser's reply; with EVAL (and plain DEL) the releaser's reply comes first, so in an event loop the releaser wakes first and its next SET NX wins before the remote waiter hears of the release. (The instaque fast path itself is unchanged since 0.10.1; 0.10.1 was fair because of GETDEL's write order.) Measured (bin/order.py, 200 rounds, notification-first): getdel 200/200, eval 0/200, del 0/200, PING pipelined before the EVAL 200/200.

Impact: with two managers in one event loop a remote waiter stalled ~3.8-4.4 s across 1000 back-to-back modifies (0.10.1: 0.01-0.04 s), oplock on or off, redis-py 7 or 8. In a real prod app (2 workers, 3000-iteration modify_state loop on one worker) the worst latency of other-worker requests went from 14-24 ms (0.10.1) to 77-165 ms (0.11.0a1); medians unchanged; final counts always exact. A mild tail-latency regression under a tight loop, not starvation in practice.

Fix

_send_lock_release sends PING and the unchanged release EVAL in one non-transactional pipeline, restoring 0.10.1's reply order — no extra round trip (uncontended modify_state ~1.15 ms/op before and after). The cancellation re-send under shield goes through the same helper. Mutual exclusion, fencing, leases and cancellation cleanup are untouched (same script).

Test plan

  • New test_remote_waiter_not_passed_over_by_back_to_back_modifies[oplock False/True] (real Redis only; skipped on the mock): before the fix 200/200 loop iterations finished before the remote waiter got in (5 runs × both modes); after, 6-7/200 in every run.
  • test_cancel_while_releasing_lock_releases_it now patches _send_lock_release instead of redis.eval; the mock pipeline gained ping / eval.
  • tests/units/istate: 269 (mock) / 270 (real Redis) / 270 (real Redis + oplock) passed. Full unit suite: 15634 passed on the mock, 15650 on real Redis, 81.4-81.5% (Playwright browser-driver module excluded locally for lack of a Chromium binary). ruff, pyright: clean.
  • Before/after (in-process max wait for the remote waiter, 3 trials): 0.11.0a1 3.8-4.2 s → this branch 0.01-0.02 s (oplock on), 0.008-0.009 s (off). Real app worst other-worker latency: 77-165 ms → 11-25 ms.

Notes

  • This relies on Redis's (undocumented) reply write order, as 0.10.1 did by accident; verified on Redis 7.0.15. With io-threads, proxies, Valkey, KeyDB or Dragonfly the order may differ, and behaviour then falls back to today's (no correctness risk). A real fair hand-off (per-waiter queue) is a larger protocol change — follow-up issue.

News fragment: news/+redis-lock-release-wakes-waiters-first.bugfix.md.

Generated by Claude Code