Orphaned Lock Reapers

Relevant source files

The following files were used as context for generating this wiki page:

• lib/sidekiq_unique_jobs/orphans/reaper.rb
• lib/sidekiq_unique_jobs/server.rb
• lib/sidekiq_unique_jobs/config.rb
• lib/sidekiq_unique_jobs/timer_task.rb
• spec/sidekiq_unique_jobs/orphans/reaper_spec.rb
• CLAUDE.md

The Orphaned Lock Reaper system is a critical maintenance component of the sidekiq-unique-jobs gem. Its primary purpose is to identify and remove "orphaned" locks—Redis keys representing job uniqueness that persist after a job has been completed, lost, or if a process crashed without proper cleanup. By cleaning these stale digests, the reaper ensures that future jobs with the same arguments are not permanently blocked by abandoned locks.

The system is designed to be safe and non-blocking. It operates by cross-referencing existing lock digests against active Sidekiq queues, retry sets, scheduled sets, and currently executing processes. It uses a "fail-closed" approach, meaning if it cannot definitively prove a lock is an orphan (e.g., during high load or network issues), it preserves the lock to prevent duplicate job execution.

Sources: lib/sidekiq_unique_jobs/orphans/reaper.rb:10-15, CLAUDE.md:67-72

Architecture and Lifecycle

The reaper system consists of three main roles: the Reaper (the worker that performs cleanup), the Manager/Server (which coordinates the lifecycle), and the Resurrector (which ensures a reaper is always active).

Coordination and Mutex

To prevent multiple Sidekiq processes from reaping simultaneously—which would cause unnecessary Redis load—the system uses a Redis-based mutex (UNIQUE_REAPER). Only one process can register as the active reaper at a time using a SET NX EX operation.

The diagram shows how the server decides whether to become the active reaper or a passive observer (resurrector).
Sources: lib/sidekiq_unique_jobs/server.rb:64-90, lib/sidekiq_unique_jobs/server.rb:116-122

The Resurrector

Every Sidekiq process that is not the active reaper runs a "Resurrector" task. This task periodically checks if the UNIQUE_REAPER mutex key has expired. If the key is missing (implying the previous reaper process died or shut down), the resurrector attempts to promote its process to be the new active reaper.

Sources: lib/sidekiq_unique_jobs/server.rb:98-112, UPGRADING.md:83-87

Reaper Logic and Discovery

The SidekiqUniqueJobs::Orphans::Reaper class implements the core discovery logic. It utilizes a Ruby-based approach (default) to ensure O(1) memory usage per digest and avoids blocking Redis operations.

Orphan Identification Process

The reaper follows a specific hierarchy when validating if a lock is orphaned:

1. Candidate Selection: It pulls digests from the uniquejobs:digests sorted set that are older than a configured grace period.
2. Liveness Check: It checks if a matching :LOCKED hash exists in Redis. If the hash is empty or missing, the digest is an orphan.
3. Sidekiq Search: It searches for the digest in:
   • The Scheduled set (sscan)
   • The Retry set (sscan)
   • All Queues (lrange in pages)
   • Active Processes (checking the :work hashes of all live Sidekiq nodes)

The sequence demonstrates the short-circuiting logic where the reaper stops searching as soon as it finds a match.
Sources: lib/sidekiq_unique_jobs/orphans/reaper.rb:45-80, lib/sidekiq_unique_jobs/orphans/reaper.rb:102-125

Safety Mechanisms

• Timeout: The reaper will stop its current cycle if it exceeds config.reaper_timeout to prevent long-running maintenance tasks from impacting performance.
• Queue Length Limit: If total enqueued jobs exceed MAX_QUEUE_LENGTH (default 1,000), the reaper skips queue scanning. This is a "fail-closed" safety to prevent expensive O(N) scans during traffic spikes.
• Grace Period: Only locks older than 10 seconds (plus reaper timeout) are considered candidates to avoid deleting locks for jobs currently being enqueued.

Sources: lib/sidekiq_unique_jobs/orphans/reaper.rb:24-26, lib/sidekiq_unique_jobs/orphans/reaper.rb:147-158, spec/sidekiq_unique_jobs/orphans/reaper_spec.rb:111-122

Configuration

The reaper behavior is governed by several global configuration options.

Option	Type	Default	Description
reaper	Symbol/Boolean	:ruby	The reaper implementation to use. Options: :ruby, :lua, :none.
reaper_interval	Integer	600	Seconds between reaper runs.
reaper_count	Integer	1000	Maximum number of digests to examine per reaper cycle.
reaper_timeout	Integer	10	Maximum duration (in seconds) the reaper is allowed to run.

Sources: README.md:94-98, lib/sidekiq_unique_jobs/config.rb:45-55

Implementation Varieties

While the Ruby reaper is the recommended default, the system supports different modes:

Ruby Reaper (:ruby)

• Pros: Memory efficient (O(1)), non-blocking to Redis, safe "fail-closed" logic.
• Cons: Slower than Lua for very large sets of orphans.
• File: lib/sidekiq_unique_jobs/orphans/reaper.rb

Lua Reaper (:lua)

• Pros: Extremely fast as it runs inside Redis.
• Cons: Can block Redis if reaper_count is set too high; less sophisticated liveness checking compared to the Ruby version.
• Guideline: Keep reaper_count ≤ 1000 when using this mode.

Sources: CLAUDE.md:67-72, CLAUDE.md:162-164

The Orphaned Lock Reaper ensures the long-term health of the unique jobs system by reconciling the state of Redis locks with the actual state of Sidekiq's workload. By utilizing a distributed mutex and thorough liveness checks, it balances the need for consistency with the performance requirements of high-volume Sidekiq clusters.