Tensor Operations & Kernels
Relevant source files
The following files were used as context for generating this wiki page:
Tensor Operations and Kernels provide the mathematical foundation for LLM inference within the oxidize workspace. This system executes matrix-vector multiplication (GEMV), quantization, and hardware-specific optimizations. The compute layer prioritizes high-performance CPU inference using SIMD intrinsics and hand-tuned kernels.
Sources: PRD.md:14-25, CLAUDE.md:15-30
Architecture Overview
The compute architecture separates high-level tensor abstractions from low-level hardware kernels. oxidize-core defines the Tensor structure and manages the compute graph. Hardware backends, such as CUDA and Metal, implement the ComputeBackend trait to provide specialized execution paths.
Sources: CLAUDE.md:62, PRD.md:105-110
The diagram shows the dispatch flow from the core API to specific hardware kernels.
Sources: CLAUDE.md:65, oxidize-core/src/compute/AGENTS.md:21
Tensor Implementation
The Tensor implementation resides in oxidize-core/src/compute/tensor/mod.rs. This module contains over 5,000 lines of code and 135 unsafe blocks for performance. It defines the Tensor struct, which includes shape, strides, and data type (DType) metadata.
Sources: CLAUDE.md:65, oxidize-core/src/compute/AGENTS.md:12
Data Types and Quantization
The system supports multiple floating-point and quantized data types. Quantization reduces model size while maintaining numerical accuracy.
| Type | Description |
|---|---|
F32 | 32-bit floating point (standard) |
F16 | 16-bit floating point (low-risk reduction) |
Q4_0 | 4-bit quantization with scales |
Q8_0 | 8-bit quantization |
Q4_K | K-quantization variant (block-wise) |
Sources: PRD.md:80-95, oxidize-kernels/AGENTS.md:25
CPU Kernels (OXK)
The oxidize-kernels (OXK) crate provides zero-dependency, hand-tuned CPU kernels. These kernels focus on Matrix-Vector Multiplication (GEMV) for quantized formats. OXK implements a "bit-exact" requirement, meaning every SIMD variant must match the scalar reference implementation precisely.
Sources: oxidize-kernels/AGENTS.md:7-15
SIMD Dispatch
The simd.rs module manages runtime detection of CPU features. It selects the optimal instruction set based on the available hardware.
- AVX-512: Highest performance for modern Intel/AMD processors.
- AVX2: Standard acceleration for x86_64 hardware.
- NEON: Acceleration for ARM64 (Apple Silicon, Graviton).
- Scalar: Fallback for legacy hardware.
Sources: oxidize-core/src/compute/AGENTS.md:21, PRD.md:110-115
Core Kernels and Constants
OXK defines specific constants for block sizes used in quantization math.
| Constant | Value | Description |
|---|---|---|
QK_K | 256 | Block size for K-quants |
BLOCK_Q4_K_SIZE | 144 | Size in bytes for Q4_K block |
BLOCK_Q8_K_BYTES | 292 | Size in bytes for Q8_K block |
Sources: oxidize-kernels/AGENTS.md:27
Hardware Detection and Autotuning
The oxidize-cpp and oxidize-core components perform hardware inventory to optimize thread allocation and NUMA (Non-Unified Memory Access) strategies.
Sources: oxidize-cpp/src/autotune.cpp:10-50, CLAUDE.md:75-80
The sequence diagram illustrates the inventory gathering and rule application during autotuning.
Sources: oxidize-cpp/src/autotune.cpp:55-100
Tuning Rules
The system follows specific logic for CPU thread allocation:
- If a model exceeds 192 GiB or 80% of RAM, it uses
NUMA interleaveand caps threads at 48. - If a model fits a single NUMA node, it uses
NUMA singleand pins threads to logical cores on node 0. - If 2 MiB hugepages are available and the model exceeds 200 GiB, it enables hugepage mmap.
Sources: oxidize-cpp/src/autotune.cpp:80-120, CLAUDE.md:165-175
Conclusion
Tensor Operations and Kernels in oxidize utilize a tiered execution model. High-level tensor logic in oxidize-core coordinates with specialized OXK kernels to provide bit-exact, SIMD-accelerated inference. This structure allows the engine to maintain high performance across diverse CPU architectures while ensuring numerical parity with reference implementations.
Sources: oxidize-kernels/AGENTS.md:38, PRD.md:120-130