Hardware Compute Backends

Relevant source files

The following files were used as context for generating this wiki page:

• oxidize-core/src/backend.rs
• oxidize-core/src/backends/cuda.rs
• oxidize-core/src/backends/metal.rs
• oxidize-core/src/backends/AGENTS.md
• oxidize-core/src/lib.rs
• CLAUDE.md
• PRD.md

Hardware compute backends provide accelerated execution paths for tensor operations by interfacing with hardware-specific APIs. The system abstracts different acceleration targets—including NVIDIA CUDA, Apple Metal, Vulkan, and WebGPU—behind a unified Rust interface to ensure consistent inference behavior across different platforms.

Sources: oxidize-core/src/backends/AGENTS.md, PRD.md:23-35

Backend Architecture

The architecture relies on a central abstraction, the ComputeBackend trait, which all specific hardware implementations must fulfill. This design decouples the high-level model logic from low-level GPU or accelerator execution details.

The ComputeBackend Trait

The ComputeBackend trait defines the standard interface for memory management and kernel execution. Every backend must implement the following core capabilities:

• Memory Allocation: Reserving space on the device for tensors.
• Data Transfer: Copying data between the host (CPU) and the device (GPU).
• Kernel Execution: Launching hardware-specific compute kernels for operations like matrix-vector multiplication (GEMV).

Sources: oxidize-core/src/backend.rs, CLAUDE.md:65, oxidize-core/src/backends/AGENTS.md

Execution Flow

When the inference engine requires a tensor operation, it dispatches the request through the ComputeBackend interface. The specific backend implementation handles the asynchronous execution and synchronization required by the hardware.

The diagram above shows the dispatch flow from the central inference engine through the abstraction trait to specific hardware execution paths.
Sources: oxidize-core/src/backends/AGENTS.md, oxidize-core/src/lib.rs:70-135

Implementation Details by Backend

NVIDIA CUDA

The CUDA backend manages NVIDIA GPU acceleration using the cudarc crate. It features asynchronous stream management to overlap data transfers with compute tasks.

• Kernel Loading: Uses cudarc for dynamic loading and management of PTX kernels.
• Synchronization: Manages async stream synchronization to ensure data integrity during parallel execution.
• Configuration: Initializations default to device index 0, though the system is designed to eventually support CUDA_VISIBLE_DEVICES.

Sources: oxidize-core/src/backends/cuda.rs, oxidize-core/src/backends/AGENTS.md

Apple Metal and MLX

These backends target Apple Silicon (macOS and iOS).

• Metal: Utilizes MTLDevice and MTLCommandBuffer for low-level GPU access. It includes pipeline caching to avoid recompiling shaders at runtime.
• MLX: A specialized framework for Apple hardware, heavily gated behind #[cfg(target_os = "macos")]. Unlike Vulkan, it does not use a stub pattern for fallbacks.

Sources: oxidize-core/src/backends/metal.rs, oxidize-core/src/backends/AGENTS.md, oxidize-core/src/lib.rs:105-110

Vulkan and WebGPU

• Vulkan: Implements compute shaders for cross-platform GPU support. It uses a "stub twin" pattern (vulkan_stub.rs) to provide a no-op fallback when the Vulkan feature is not compiled.
• WebGPU: Targets browser-based inference and WASM environments via wgpu, utilizing compute passes for execution.

Sources: oxidize-core/src/backends/AGENTS.md, oxidize-core/src/lib.rs:125-135

Build Configuration and Feature Gates

Backends are enabled via Cargo features. This allows the workspace to maintain a light dependency footprint by only compiling the necessary hardware drivers.

Feature	Backend Implementation	Target Platforms
cuda	NVIDIA CUDA via cudarc	Linux, Windows
metal	Apple Metal	macOS, iOS
vulkan	Vulkan Compute	Cross-platform
webgpu	WebGPU / wgpu	WASM, Browser
mlx	Apple MLX Framework	macOS-only
strix	Custom Accelerator	Strix Hardware

Sources: oxidize-core/src/backends/AGENTS.md, oxidize-core/Cargo.toml:16-25

Technical Conventions

Build Info Micro-Pattern

To facilitate runtime and compile-time detection of hardware support, every backend exposes a standard "Build Info" pattern.

• Struct: XxxBuildInfo (e.g., CudaBuildInfo).
• Function: xxx_build_info() (e.g., cuda_build_info()).
  These helpers allow the autotune module to query available hardware and select the optimal execution plan.

Sources: oxidize-core/src/backends/AGENTS.md, oxidize-core/src/lib.rs:145-160

Error Handling and Safety

GPU backends frequently involve FFI (Foreign Function Interface) and raw pointer manipulation.

1. Safety Isolation: unsafe blocks are strictly isolated within the backend-specific files (cuda.rs, metal.rs).
2. Error Chaining: Hardware-specific errors are wrapped into the central OxidizeError or anyhow::Result types used by the core library.

Sources: oxidize-core/src/backends/AGENTS.md, CLAUDE.md:120-125

Data Flow: Host-to-Device Execution

The following sequence diagram illustrates the typical lifecycle of a compute task through a hardware backend.