Batch Delete Utilities
Relevant source files
The following files were used as context for generating this wiki page:
Batch Delete Utilities in sidekiq-unique-jobs provide efficient mechanisms for removing large numbers of unique job digests and their associated locks from Redis. These utilities are designed to minimize Redis overhead by leveraging pipelining and slicing large collections into manageable chunks. This is critical for maintaining performance when clearing queues, handling job expirations, or performing version upgrades.
These utilities are utilized across various system components including the Command Line Interface (CLI), the Web UI, and the internal UpgradeLocks system.
Sources: lib/sidekiq_unique_jobs/batch_delete.rb, spec/sidekiq_unique_jobs/batch_delete_spec.rb
Core Architecture
The primary component for these operations is the SidekiqUniqueJobs::BatchDelete class. It encapsulates the logic for iterating over a collection of digests and removing both the digest entry from the central sorted set and the specific metadata hash associated with that digest.
Batch Processing Logic
To ensure stability and performance, the utility uses a defined BATCH_SIZE (defaulting to 500) to slice the input array. It uses Redis pipelining to group commands, reducing the number of network round-trips.
The diagram shows the iterative process of slicing digests and using Redis pipelines for atomic-like deletion of hashes and set members.
Sources: lib/sidekiq_unique_jobs/batch_delete.rb:14-87
Components and Constants
| Component | Description | Source |
|---|---|---|
BATCH_SIZE | Default size of 500 for slicing digest collections to prevent Redis blocking. | lib/sidekiq_unique_jobs/batch_delete.rb:14 |
del_digest | Internal method that specifically uses UNLINK on the #{digest}:LOCKED key. | lib/sidekiq_unique_jobs/batch_delete.rb:90-92 |
SidekiqUniqueJobs::DIGESTS | The constant representing the Redis sorted set key (uniquejobs:digests) where all digests are stored. | lib/sidekiq_unique_jobs/batch_delete.rb:84 |
Integration Points
Command Line Interface (CLI)
The CLI provides a del command that allows users to delete unique digests by pattern. It internally triggers batch deletion logic. The CLI supports a --dry-run option to simulate deletions without actually removing keys from Redis.
Sources: spec/sidekiq_unique_jobs/cli_spec.rb:17-104
Web Interface
The Sidekiq Web UI extension uses these utilities to allow administrators to "Delete All" locks or delete specific locks through a browser. The SidekiqUniqueJobs::Web::Helpers module provides the bridge between the UI actions and the underlying Digests collection management.
Sources: spec/sidekiq_unique_jobs/web_spec.rb:80-106, lib/sidekiq_unique_jobs/web/helpers.rb:43-45
Upgrade and Maintenance
During gem version upgrades (e.g., v8 to v9), the UpgradeLocks utility uses batch scanning and deletion to remove obsolete Redis keys such as :QUEUED, :PRIMED, and :INFO suffixes that are no longer used in the newer architecture.
This sequence illustrates how maintenance tasks leverage batch deletion to clean up legacy keys during system upgrades.
Sources: lib/sidekiq_unique_jobs/upgrade_locks.rb:112-167
Implementation Details
The utility is designed to be connection-aware, allowing it to accept an existing Redis connection or fetch one from the pool.
# lib/sidekiq_unique_jobs/batch_delete.rb:76-88
def batch_delete(conn)
digests.each_slice(BATCH_SIZE) do |chunk|
conn.pipelined do |pipeline|
chunk.each do |digest|
del_digest(pipeline, digest)
pipeline.zrem(SidekiqUniqueJobs::DIGESTS, digest)
@count += 1
end
end
end
@count
end
Sources: lib/sidekiq_unique_jobs/batch_delete.rb:76-88
Summary
Batch Delete Utilities provide the necessary performance optimizations for large-scale data removal in sidekiq-unique-jobs. By combining chunked processing, Redis pipelining, and the use of non-blocking UNLINK commands, the system ensures that maintenance and administrative tasks do not degrade the performance of the Sidekiq job processing pipeline.