What is sidekiq-unique-jobs?
Relevant source files
The following files were used as context for generating this wiki page:
sidekiq-unique-jobs is a Sidekiq middleware gem designed to prevent duplicate jobs from being enqueued or executed. It leverages Redis-based locking mechanisms to ensure that only one instance of a job with specific arguments exists within the Sidekiq ecosystem at any given time, based on configurable rules.
Sources: README.md:3-5, CLAUDE.md:6-8
Core Architecture and Middleware
The system operates primarily through two middleware components that intercept Sidekiq's lifecycle:
- Client Middleware: Intercepts the job pushing process to prevent duplicate enqueuing.
- Server Middleware: Intercepts job execution to prevent concurrent processing of the same job.
Data Flow for Job Enqueuing
When a job is pushed, the gem generates a unique lock_digest based on the worker class, queue, and arguments. It then attempts to acquire a lock in Redis before allowing the job to be added to the Sidekiq queue.
Lock Types
The gem provides several lock strategies that control the boundaries of uniqueness:
| Type | Stage | Description |
|---|---|---|
:until_executing | Client | Locks on enqueue; unlocks just before the job starts performing. |
:until_executed | Client | Locks on enqueue; unlocks only after the job completes successfully. |
:until_expired | Client | Locks on enqueue; remains locked until a configured TTL expires. |
:while_executing | Server | Prevents concurrent execution of the same job; no enqueue protection. |
:until_and_while_executing | Both | Combines enqueue protection and concurrent execution protection. |
Sources: README.md:46-52, CLAUDE.md:18-24
Conflict Resolution Strategies
When a duplicate job is detected (a "conflict"), the gem applies a strategy to handle the incoming duplicate:
:log: The default strategy. It logs the occurrence and silently discards the duplicate job.:raise: Raises aSidekiqUniqueJobs::OnConflict::Raiseerror.:reject: Moves the duplicate job directly to the Sidekiq Dead Set.:replace: Deletes the existing job/lock and enqueues the new one.:reschedule: Schedules the duplicate job to run at a later time.
Sources: README.md:56-65, CLAUDE.md:43-50
Lock Lifecycle and Maintenance
Key Management
In version 9+, the gem has optimized Redis storage, using primarily two keys per lock: a digest:LOCKED hash for lock metadata and a uniquejobs:digests sorted set for tracking active digests.
Orphan Cleanup (The Reaper)
To prevent stale locks from occupying Redis indefinitely (e.g., after a process crash), the gem includes a "Reaper" system. It periodically scans for orphaned locks where a lock exists in Redis but the corresponding job is no longer in any Sidekiq queue or retry set.
Configuration
Global configuration is managed via a configure block, while per-worker overrides are defined in sidekiq_options.
Global Configuration Options
| Option | Default | Description |
|---|---|---|
lock_ttl | nil | Expiration time for locks in seconds. |
lock_timeout | 0 | Seconds to wait for a lock before failing (0 = immediate). |
enabled | true | Globally enable or disable uniqueness. |
reaper | :ruby | Strategy for cleaning orphans (:ruby or :lua). |
digest_algorithm | :legacy | Hash algorithm: :legacy (MD5) or :modern (SHA3-256). |
Sources: README.md:73-85, lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb:162-171
Per-Worker Options
sidekiq_options lock: :until_executed,
lock_ttl: 3600,
unique_across_queues: true,
lock_args_method: :custom_args
Sources: README.md:94-110, CLAUDE.md:113-117
Monitoring and Observability
The gem provides two primary ways to monitor locks:
- Web UI: An extension for the Sidekiq Web interface that adds a "Locks" tab for browsing and manual deletion of locks.
- Reflections: A hook system that allows developers to observe events (like
locked,unlocked, orlock_failed) for logging or metrics.
SidekiqUniqueJobs.reflect do |on|
on.locked { |job| Rails.logger.info("Job locked: #{job['jid']}") }
end
Sources: README.md:129-133, 153-160, lib/sidekiq_unique_jobs/web/helpers.rb:44-48, lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb:271-283
Summary
sidekiq-unique-jobs provides a robust framework for managing job idempotency and concurrency in Sidekiq. By combining various lock types, conflict strategies, and a dedicated cleanup reaper, it ensures Redis remains consistent while preventing duplicate business logic execution.