Performance Benchmarking
Relevant source files
The following files were used as context for generating this wiki page:
Performance benchmarking in sidekiq-unique-jobs is a structured process used to measure the efficiency of locking mechanisms, identify regressions across different git branches, and optimize Redis interactions. The system focuses on high-level metrics such as Iterations Per Second (IPS) and memory allocation, as well as granular Redis metrics like command counts per lock cycle.
The benchmarking suite is essential for validating optimizations in core components like the Orphan Reaper, Lua script execution, and lock acquisition strategies. By comparing metrics between the main branch and feature branches, developers ensure that new features do not introduce latency or excessive memory overhead.
Benchmark Tooling and Scripts
The project provides several specialized scripts to handle different benchmarking needs, ranging from full suite execution to specific implementation comparisons.
Available Scripts
| Script | Purpose | Key Metrics |
|---|---|---|
bin/benchmark | Comprehensive testing of all gem aspects. | Lock acquisition, contention, digest generation, memory usage. |
bin/bench | Focused benchmarking for the Orphan Reaper. | Ruby vs. Lua implementation performance. |
bin/benchmark_improvements | Tracks specific performance optimizations. | Non-blocking acquisition, TTL calculation, thread-safe operations. |
bin/compare_performance | Automated comparison between git branches. | Summary of relative performance changes (e.g., % improvement). |
autoresearch.sh | Deep Redis-level analysis. | Redis commands per cycle, keys created, throughput. |
Sources: BENCHMARKING.md:7-40, autoresearch.sh:9-14
Branch Comparison Workflow
The bin/compare_performance tool automates the validation of performance improvements by switching branches and aggregating results.
The comparison tool preserves the developer's working state while providing a clear "before and after" view of performance metrics.
Sources: BENCHMARKING.md:33-46
Core Performance Metrics
The benchmarking suite evaluates the system based on three primary categories: throughput, memory, and Redis efficiency.
1. Iterations Per Second (IPS)
This measures how many operations (lock/unlock) the system can perform per second. A higher value indicates better performance. It is used to compare different lock flows, such as "normal flow" versus "failure simulation".
Sources: BENCHMARKING.md:75-84
2. Memory Usage
Benchmarks track both allocated memory (total memory used during the operation) and retained memory (memory that remains in use after the operation finishes). Lower values are preferred to prevent memory bloat in high-concurrency environments.
Sources: BENCHMARKING.md:92-98
3. Redis Efficiency (Autoresearch)
The autoresearch.sh script provides low-level insights into how the gem interacts with Redis. It uses INFO commandstats to count the number of Redis calls per lock lifecycle.
This flow allows developers to see exactly how many Redis operations a specific lock type consumes.
Sources: autoresearch.sh:50-70
Tracked Optimizations
The bin/benchmark_improvements script specifically monitors several key architectural optimizations:
- Non-Blocking Lock Acquisition: Reducing
waittime from 1 second to 0 in exception handling to prevent worker blocking. - Orphan Reaper Optimization: Using Lua-based cached digest transformations and short-circuit breaks to improve cleanup speed by an estimated 20-40%.
- Thread-Safe Config: Implementing mutexes with double-check locking for Redis version fetching to prevent race conditions.
- Redis Operation Capping: Limiting blocking waits to 5 seconds (
MAX_BLOCKING_WAIT) to prevent indefinite client middleware hangs.
Sources: BENCHMARKING.md:104-126, CLAUDE.md:88-92
Lock Metrics Integration
The system includes a LockMetrics class that tracks events in Redis for observability. These metrics are integrated into the benchmarking process to verify that lock events (locked, unlocked, failed) are occurring as expected.
Metric Tracking Logic
| Event | Tracking Trigger |
|---|---|
locked | Recorded upon successful client-side or server-side lock acquisition. |
lock_failed | Recorded when a lock is already held by another JID. |
unlocked | Recorded when the middleware successfully releases the digest. |
Sources: spec/sidekiq_unique_jobs/lock_metrics_integration_spec.rb:14-60, spec/sidekiq_unique_jobs/lock_metrics_spec.rb:10-20
Lock Metrics Data Flow
Metrics are accumulated in memory and then flushed to Redis buckets, typically organized by time (e.g., minute buckets).
Continuous Monitoring and CI Integration
Performance benchmarking is integrated into the CI/CD pipeline via GitHub Actions. This ensures that every Pull Request is evaluated for performance regressions before being merged into the main branch.
Benchmarking Prerequisites
To ensure accurate results, benchmarks should be run under consistent conditions:
- Quiet Machine: Minimize background application noise.
- Consistent Redis: Use the same Redis version (e.g., 6.2+ for LMOVE support).
- Warm-up: Run multiple iterations to allow for JIT compilation.
Sources: BENCHMARKING.md:144-150, README.md:15-20
Performance benchmarking in this project provides the empirical data necessary to maintain a high-performance middleware. By combining iterative throughput testing with low-level Redis command analysis, the project ensures that the locking logic remains efficient even as complexity grows.