Core Inference Engine Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Core Inference Engine provides high-performance LLM (Large Language Model) execution primitives within the oxidize-core crate. It manages model loading, tensor operations, quantization, and token generation across multiple hardware backends including CPU, CUDA, Metal, and WASM.
Sources: oxidize-core/src/lib.rs, PRD.md:28-40
Architecture Overview
The inference engine follows a layered architecture that separates model definitions from hardware-specific compute kernels. oxidize-core centralizes inference logic to ensure consistent behavior across CLI, server, and language bindings.
The diagram shows the request flow from interface crates to the core engine and down to hardware acceleration backends.
Sources: CLAUDE.md:21-45, PRD.md:28-40
Model Abstraction
The engine defines a Model trait as the primary boundary for all inference implementations. This abstraction allows different model types, such as standard inference, speculative decoding, or layer-wise caching, to be swapped interchangeably.
Core Model Trait
The Model trait requires implementations to provide:
forward: Processes tokens and updates session state.forward_many: Batch processes tokens.vocab_size: Returns the total vocabulary count.context_size: Returns the maximum sequence length.layer_count: Returns the number of transformer layers.
Sources: oxidize-core/src/model/AGENTS.md:30-34, PRD.md:156-163
Model Implementations
The engine includes five primary Model implementations:
| Struct | Purpose |
|---|---|
Inference | The standard implementation for unified inference. |
Llama | Optimized path for Llama-family architectures. |
LayerWise | Implements an LRU cache for layers to support large models on limited memory. |
MLX | macOS-specific implementation using the MLX framework. |
DFlash | A draft model implementation optimized for speculative decoding. |
Sources: CLAUDE.md:82, oxidize-core/src/model/AGENTS.md:9-16
Configuration and Metadata
Model behavior is controlled by the InferenceConfig (Rust) or OcInferenceConfig (C). This configuration mirrors the metadata found in GGUF files and defines the transformer geometry and positional encoding parameters.
Inference Parameters
| Parameter | Type | Description |
|---|---|---|
vocab_size | u32 | Total tokens in vocabulary. |
hidden_size | u32 | Dimensionality of hidden layers. |
num_attention_heads | u32 | Number of query heads. |
num_key_value_heads | u32 | Number of KV heads (for GQA/MQA). |
rms_norm_eps | f32 | Epsilon for normalization stability. |
rope_theta | f32 | Base frequency for Rotary Position Embeddings. |
sliding_window | u32 | Size of SWA window (0 for global attention). |
Sources: oxidize-cpp/include/oxidize/config.hpp:51-86, oxidize-c/include/oxidize/inference.h:32-75
Execution Flow
The generation process involves a sequence of tokenization, prefill, and autoregressive decoding. The Session struct tracks state, including consumed tokens and KV cache pointers, during the generation loop.
The sequence shows how the engine orchestrates backends and sessions to generate text.
Sources: oxidize-core/src/model/AGENTS.md:36-39, PRD.md:280-285
Hardware Backends and Acceleration
The ComputeBackend trait abstracts hardware-specific implementations. Backends are responsible for tensor operations like Matrix-Vector multiplication (GEMV) and Attention.
- CPU: Uses SIMD kernels (AVX2, AVX512, NEON) for high-performance local execution.
- CUDA: Provides NVIDIA GPU acceleration with kernel fusion and Flash Attention support.
- Metal: Optimized for Apple Silicon unified memory architecture.
- WASM/WebGPU: Enables browser-based inference.
Sources: CLAUDE.md:81, PRD.md:105-144, oxidize-core/src/lib.rs:88-154
Summary
The Core Inference Engine Architecture enables cross-platform LLM deployment by centralizing model logic in a modular Rust core. By utilizing standardized traits for models and backends, it supports a wide variety of architectures (Llama, Mistral, Gemma, etc.) and quantization formats while maintaining high execution efficiency through hardware-specific optimizations.