Model Formats: GGUF & SafeTensors
Relevant source files
The following files were used as context for generating this wiki page:
Oxidize supports two primary model formats for Large Language Model (LLM) inference: GGUF and SafeTensors. The project provides infrastructure for parsing, loading, and converting these formats to ensure high-performance execution across different hardware backends.
The oxidize-core crate manages the core logic for format parsing and tensor information extraction. The oxidize-convert utility enables users to transform HuggingFace SafeTensors into the GGUF format, which is optimized for local inference and quantization.
Sources: oxidize-core/src/format/AGENTS.md, oxidize-core/src/lib.rs:77-85, PRD.md:158-160
GGUF (Georgi Gerganov Universal Format)
GGUF serves as the primary format for inference in Oxidize. It stores model weights, quantization types, and comprehensive metadata in a single file optimized for memory-mapped I/O.
Architecture and Data Structures
The GGUF implementation centers around the GgufQuantizationType enum, which acts as the central hub for quantization format constants used across the workspace.
The GGUF parser validates magic numbers, handles versions (v2 and v3), and extracts tensor metadata including shape, strides, and alignment.
Sources: oxidize-core/src/format/gguf.rs, oxidize-core/src/format/AGENTS.md, PRD.md:65-73
Key Components
| Component | Description |
|---|---|
GgufModelLoader | Implements the ModelLoader trait to parse GGUF files and extract mapped tensor info. |
GgufMetadataValue | Enumeration supporting multiple types: String, Bool, varying integer widths, floats, and arrays. |
GgufQuantizationType | The source of truth for quantization types, including standard Q-quants and custom AL-family quants (AL5, AL8). |
Sources: oxidize-cli/src/commands.rs:9-15, oxidize-core/src/format/AGENTS.md, AGENTS.md:325-330
SafeTensors
SafeTensors is a format designed by HuggingFace for safe and fast storage of tensors. Oxidize implements memory-mapped loading for SafeTensors to facilitate efficient weight extraction and conversion.
Features
- Memory Mapping: The format supports zero-copy loading using
memmap2. - Safety: Validates headers to prevent arbitrary code execution during model loading.
- Metadata: Stores tensor shapes and data types (DType) compatible with the Oxidize compute layer.
Sources: oxidize-core/src/format/safetensors.rs, oxidize-core/src/format/AGENTS.md, PRD.md:75-77
Format Conversion: SafeTensors to GGUF
The oxidize-convert crate provides logic to transform SafeTensors checkpoints into GGUF files. This process involves mapping HuggingFace tensor names to the naming convention used by GGUF architectures (e.g., Llama, Qwen).
Conversion Workflow
The conversion process flows from raw SafeTensors input through a mapping layer that resolves metadata and architecture-specific tensor names.
Key functions in this workflow include map_hf_tensor_name() and convert_safetensors_to_gguf(). The conversion utility also supports optional pruning during the transformation process.
Sources: oxidize-convert/src/main.rs, oxidize-core/src/format/AGENTS.md, AGENTS.md:120-125
Usage and Integration
Oxidize provides CLI tools to inspect and manage these formats. The show and inspect commands allow developers to view metadata and tensor layouts directly from the terminal.
CLI Inspection Commands
oxidize show <model>: Rreadable summary of model name, architecture, quantization, and context length.oxidize inspect <model>: Detailed list of every metadata key and tensor dimension/offset.
Sources: oxidize-cli/src/commands.rs:9-62, oxidize-cli/src/commands.rs:75-96
Quantization Targets
When converting or quantizing models, the following common targets are supported:
| Target | Risk/Benefit |
|---|---|
F16 | Low-risk size reduction (50% of F32). |
Q8_0 | 8-bit quantization with minimal quality loss. |
Q4_0 / Q4_K_M | 4-bit quantization for significant memory savings. |
AL5 | MSE-optimized 4-bit custom quantization. |
Sources: oxidize-cli/src/commands.rs:90-95, AGENTS.md:328-332, README.md:118-125
Summary
Oxidize centralizes model format logic in the oxidize-core format module. By supporting GGUF v3 and SafeTensors, the system enables high-performance inference through memory-mapped I/O and versatile quantization schemes. The inclusion of the oxidize-convert utility ensures compatibility with the broader HuggingFace ecosystem while optimizing models for local-first execution.