CUDA Error Handling

Relevant source files

The following files were used as context for generating this wiki page:

• include/gnuradio/cuda/cuda_error.h
• include/gnuradio/cuda/cuda_block.h
• include/gnuradio/cuda/api.h
• include/gnuradio/cuda/cuda_block_helper.h
• lib/CMakeLists.txt

CUDA Error Handling in the gr-cuda project provides a standardized mechanism for detecting, logging, and responding to failures occurring within the CUDA Runtime and Driver APIs. The system is designed to integrate seamlessly with the GNU Radio logging framework while ensuring that GPU-related exceptions are propagated correctly to the host application.

The core of this system is a set of overloaded utility functions that evaluate CUDA return codes. If an operation fails, the system captures diagnostic information—including the error name and description—and raises a C++ exception to prevent undefined behavior in the signal processing pipeline.

Sources: include/gnuradio/cuda/cuda_error.h:11-30, include/gnuradio/cuda/cuda_block.h:20-25

Core Error Handling Mechanism

The project utilizes the check_cuda_errors function as the primary interface for error detection. This function is overloaded to support both the CUDA Runtime API (using cudaError_t) and the CUDA Driver API (using CUresult).

logic Flow

When a CUDA API call is wrapped with check_cuda_errors, the function checks the return code. If the code is anything other than cudaSuccess (or CUDA_SUCCESS), it performs the following steps:

1. Format Message: Constructs a human-readable string containing the provided context, the specific CUDA error name, and the error string provided by the NVIDIA driver.
2. Optional Logging: If a gr::logger instance is provided, it logs the error details.
3. Exception Throwing: It throws a std::runtime_error containing the formatted diagnostic message.

The diagram shows the conditional logic path taken by the error checker.
Sources: include/gnuradio/cuda/cuda_error.h:35-51

API Overloads and Signature

The error handling interface is exported via the CUDA_API macro to ensure visibility across library boundaries.

Function Signature	Purpose	Parameter: rc/res	Parameter: context	Parameter: logger
check_cuda_errors(cudaError_t rc, ...)	Handles Runtime API errors	Result of cudaMalloc, cudaStreamCreate, etc.	Description of the operation (e.g., "stream creation")	Optional gr::logger pointer
check_cuda_errors(CUresult res, ...)	Handles Driver API errors	Result of cuMemAlloc, cuDeviceGet, etc.	Description of the operation	Optional gr::logger pointer

Sources: include/gnuradio/cuda/cuda_error.h:35-51, include/gnuradio/cuda/api.h:18-24

Integration with CUDA Blocks

The cuda_block base class uses this mechanism during its lifecycle management, specifically for stream creation and destruction.

Stream Lifecycle Management

In the cuda_block constructor, cudaStreamCreateWithFlags is wrapped with an error check to ensure that the non-blocking stream is successfully allocated before any signal processing begins. If the GPU driver cannot allocate a stream, the block initialization fails immediately with a descriptive error.

// Example of error handling in block initialization
cuda_block()
{
    check_cuda_errors(cudaStreamCreateWithFlags(&d_stream, cudaStreamNonBlocking),
                      "cuda_block::cuda_block cudaStreamCreateWithFlags");
}

Sources: include/gnuradio/cuda/cuda_block.h:114-118

Synchronization and Events

While explicit error checking is common during setup, the project also handles potential GPU-side issues during execution through event-based synchronization. Functions like wait_for_work and mark_work_done interact with cuda_buffer objects, which internally rely on CUDA events (cudaEventRecord, cudaStreamWaitEvent) to manage data hazards.

The diagram illustrates how error checking acts as a gatekeeper during the block execution cycle.
Sources: include/gnuradio/cuda/cuda_block_helper.h:38-115, include/gnuradio/cuda/cuda_block.h:80-95

Build System Requirements

The error handling system relies on both the CUDA Runtime and Driver libraries. These dependencies are enforced in the CMakeLists.txt file to ensure that symbols like cudaGetErrorString and cuGetErrorName are available during linking.

target_link_libraries(
    gnuradio-cuda
    PUBLIC
        gnuradio::gnuradio-runtime
        CUDA::cudart
        CUDA::cuda_driver
        CUDA::cufft
)

Sources: lib/CMakeLists.txt:54-61

Conclusion

CUDA Error Handling in gr-cuda provides a robust safety net for GPU operations. By centralizing error detection in check_cuda_errors and integrating it into the cuda_block base class and helper functions, the project ensures that hardware failures or invalid API calls are caught early and reported with sufficient technical context for debugging.