Atomic Redis Operations with Lua

Relevant source files

The following files were used as context for generating this wiki page:

• lib/sidekiq_unique_jobs/script.rb
• lib/sidekiq_unique_jobs/script/caller.rb
• lib/sidekiq_unique_jobs/script/scripts.rb
• lib/sidekiq_unique_jobs/script/script.rb
• lib/sidekiq_unique_jobs/script/client.rb
• lib/sidekiq_unique_jobs/script/template.rb
• lib/sidekiq_unique_jobs/script/lua_error.rb
• autoresearch.md

Atomic Redis operations are a cornerstone of the sidekiq-unique-jobs gem, ensuring that lock acquisition and release remain consistent and race-condition free. By utilizing Lua scripts, multiple Redis commands are executed as a single, atomic unit on the Redis server, which prevents partial updates or inconsistent states that could occur if executed sequentially from the Ruby client.

The system handles the full lifecycle of these scripts, including loading them into Redis memory via SHA1 hashes for performance, managing script dependencies through partial templates, and providing robust error handling for Lua-specific execution failures.

Sources: CLAUDE.md:46-48, lib/sidekiq_unique_jobs/script/caller.rb:10-14, autoresearch.md:88-90

Script Execution Pipeline

The execution of a Lua script follows a structured path from the Ruby interface down to the Redis server. The Script::Caller module provides the high-level convenience methods, while the Script::Client manages retries and error handling.

The following sequence diagram illustrates how a script is called and executed:

The execution logic includes an automatic retry mechanism if Redis returns a NOSCRIPT or BUSY error. In the case of NOSCRIPT, the gem reloads the script from disk and retries the execution.
Sources: lib/sidekiq_unique_jobs/script/caller.rb:39-49, lib/sidekiq_unique_jobs/script/client.rb:42-53, lib/sidekiq_unique_jobs/script/scripts.rb:88-91

Script Arguments Normalization

Before execution, arguments (argv) are normalized. Boolean values are converted to integers (true to 1, false to 0) and nil values are converted to 0 to ensure compatibility with Lua's type system. Additionally, the gem automatically appends global metadata to every script call, including the current timestamp, debug settings, and Redis version.
Sources: lib/sidekiq_unique_jobs/script/caller.rb:57-61, lib/sidekiq_unique_jobs/script/caller.rb:113-122

Component Architecture

The Lua integration is organized into several specialized classes that handle template rendering, caching, and execution.

Component	Responsibility
Script::Caller	Entry point mixin providing the call_script convenience method.
Script::Client	Orchestrator that manages execution timing, logging, and retry logic.
Script::Scripts	A thread-safe collection (using Concurrent::Map) that caches loaded script metadata.
Script::Script	Represents a single .lua file, managing its source, path, and compiled SHA1.
Script::Template	Handles ERB-based rendering and inclusion of partial Lua files.
Script::LuaError	Specialized exception class that reformats Redis errors into readable Lua stack traces.

Sources: lib/sidekiq_unique_jobs/script/caller.rb:10, lib/sidekiq_unique_jobs/script/client.rb:8, lib/sidekiq_unique_jobs/script/scripts.rb:10, lib/sidekiq_unique_jobs/script/script.rb:10, lib/sidekiq_unique_jobs/script/template.rb:13, lib/sidekiq_unique_jobs/script/lua_error.rb:16

Script Loading and Caching

Scripts are loaded from the filesystem (defaulting to lib/sidekiq_unique_jobs/lua) and stored in a thread-safe map. Each script is rendered once using the Template engine, which supports including partials via <%= include_partial 'path' %>.
Sources: lib/sidekiq_unique_jobs/script.rb:23-26, lib/sidekiq_unique_jobs/script/scripts.rb:14-17, lib/sidekiq_unique_jobs/script/template.rb:26-29

Optimization and Atomic Operations

Recent optimizations have focused on reducing the number of Redis commands per lock lifecycle. By merging multiple logic checks and state updates into single Lua scripts, the gem achieves high throughput.

The queue_and_lock.lua Pattern

This optimized script combines multiple operations that were previously separate Redis calls:

1. Capacity Check: Uses HLEN to verify if the lock limit has been reached.
2. Digest Creation: Registers the unique digest using SET.
3. Tracking: Adds the digest to the global digests sorted set using ZADD.
4. Acquisition: Writes the job ID and timestamp to the LOCKED hash using HSET.

This consolidation reduced the command count for a lock lifecycle from 34 to 8 commands.
Sources: autoresearch.md:27-35, autoresearch.md:80-92

Resource Cleanup

The unlock.lua script handles atomic cleanup of all lock-related keys. It utilizes UNLINK to non-blockingly delete the digest, info, locked hash, and primed keys in a single operation, followed by ZREM to remove the digest from the tracking set.
Sources: autoresearch.md:100-103

Error Handling and Debugging

The gem provides a sophisticated LuaError class to handle script failures. When Redis returns a script error, it typically includes a cryptic message and a line number. LuaError intercepts these, reads the original Lua source file, and generates a formatted error context showing exactly where the script failed.

# Example of the reformatting logic
PATTERN = /ERR Error (compiling|running) script \(.*?\): .*?:(\d+): (.*)/
# Generates context like:
#  => 10: redis.call('HSET', KEYS[1], ARGV[1], ARGV[2])

Sources: lib/sidekiq_unique_jobs/script/lua_error.rb:16-44, lib/sidekiq_unique_jobs/script/lua_error.rb:55-66

Configuration for Lua Scripts

The Lua system can be configured independently or via the main SidekiqUniqueJobs config:

• debug_lua: Enables verbose output for debugging script behavior.
• max_history: Controls the length of the changelog/history stored in Redis.
• scripts_path: Pathname where .lua files are located.

Sources: lib/sidekiq_unique_jobs/script.rb:24, lib/sidekiq_unique_jobs/script/caller.rb:98-105

Summary

Atomic Redis operations via Lua are the primary mechanism for maintaining integrity in sidekiq-unique-jobs. The system manages the complexity of script lifecycle—from ERB template rendering and dependency management to SHA-based caching and advanced error reporting—ensuring that lock operations are both high-performance and reliable. By consolidating operations into scripts like queue_and_lock.lua, the gem minimizes network round-trips and maximizes Redis efficiency.