Synchronization and Stress Testing
Relevant source files
The following files were used as context for generating this wiki page:
Synchronization and stress testing in gr-cuda ensure that data integrity is maintained as it flows through high-performance GPU pipelines. The system relies on a combination of CUDA events, cuda_buffer management, and explicit synchronization helpers to prevent race conditions during host-to-device (H2D), device-to-device (D2D), and device-to-host (D2H) transfers. Sources: README.md
The testing framework uses long-running stress tests and complex flowgraphs (fan-out/fan-in) to validate that the automated synchronization mechanisms handle variable-rate production, consumer backpressure, and multiple concurrent CUDA streams without data corruption or deadlocks. Sources: python/cuda/qa_long_race_stress.py, python/cuda/qa_general_work_sync.py
Synchronization Architecture
The gr-cuda runtime manages synchronization automatically through cuda_buffer. This component handles CUDA-event synchronization between blocks, allowing GPU kernels to run back-to-back without round-tripping through the CPU. Sources: README.md
Event-Based Flow Control
Synchronization is achieved using two primary event types:
- Device-Ready Events: Signal that data is available in the buffer for a downstream consumer.
- Read-Done Events: Signal that a consumer has finished reading, making the buffer safe for the upstream producer to overwrite.
The following diagram illustrates the synchronization sequence between an upstream producer and a downstream consumer:
Synchronization Helpers
For C++ implementations, gr-cuda provides inline helper functions in cuda_block_helper.h to manage these events within the work() or general_work() functions.
| Function | Purpose | Implementation Detail |
|---|---|---|
wait_for_work | Pre-kernel sync | Calls wait_device_ready on inputs and wait_read_done on outputs. |
mark_work_done | Post-kernel sync | Calls mark_device_ready on outputs and mark_read_done on inputs. |
Sources: include/gnuradio/cuda/cuda_block_helper.h:31-118
Stress Testing Paradigms
Stress tests are designed to expose race conditions in the buffer management logic, particularly when dealing with non-1:1 rate blocks or complex topologies.
Fan-In/Fan-Out Race Stress
This test configuration validates synchronization across multiple parallel GPU branches that converge into a single GPU block (fan-in) before outputting to multiple CPU sinks (fan-out). It specifically targets "race windows" where a producer might attempt to overwrite a buffer before a consumer has finished reading.
General Work and Variable Rate Testing
Blocks using general_work() present unique synchronization challenges because the number of items consumed and produced can vary per call. Tests like test_chained_delay_blocks and test_explicit_produce_chain verify that the wrapper calls mark_work_done() before produce() advances the write pointer, which is critical for maintaining data integrity. Sources: python/cuda/qa_general_work_sync.py:157-190, include/gnuradio/cuda/cuda_block_helper.h:92-101
Key Test Components
| Class/Function | Role in Testing | Source File |
|---|---|---|
wrap_check_sink | Validates repeating sequences and tracks mismatch counts to detect sync issues. | qa_long_race_stress.py:16 |
qa_stress_sync | Tests long chains (e.g., 100+ blocks) to ensure stream synchronization across many stages. | qa_stress_sync.py:23 |
_gpu_scale_explicit_produce | Uses heavy scratch work (trigonometric loops) to widen the race window for synchronization validation. | qa_general_work_sync.py:82 |
_gpu_delay | Implements a variable delay to exercise the forecast and consume_each logic in GPU blocks. | qa_general_work_sync.py:30 |
Performance and Reliability
Stress tests utilize environment variables to scale intensity, allowing for long-duration soak testing. For example, GR_CUDA_STRESS_SECONDS (defaulting to 3600s) and GR_CUDA_STRESS_CHAIN (defaulting to 64 blocks) allow developers to push the system beyond typical usage scenarios to ensure stability. Sources: python/cuda/qa_long_race_stress.py:76-81
The integration of mark_work_done before calling produce() or consume() in general_work blocks is a critical requirement highlighted in the codebase to prevent downstream consumers from seeing stale event markers. Sources: include/gnuradio/cuda/cuda_block_helper.h:92-101, README.md:126-135
Conclusion
Synchronization in gr-cuda is a robust, event-driven system that abstracts the complexity of CUDA stream management from the user. Through extensive stress testing involving high fan-out ratios, long block chains, and variable-rate processing, the framework ensures that GPU-accelerated signal processing remains as reliable as traditional CPU-based GNU Radio implementations.