Model Quantization
Relevant source files
The following files were used as context for generating this wiki page:
Model Quantization in the oxidize workspace refers to the system for reducing the precision of LLM weights to optimize memory usage and inference speed. The project implements quantization and dequantization schemes matching llama.cpp standards, allowing large models to run on commodity hardware such as CPUs, GPUs, and Apple Silicon.
This feature spans across the core compute layer (oxidize-core), offline conversion utilities (oxidize-quantize), and specialized ports in C, C++, and Python. It supports a wide array of formats ranging from standard 16-bit floating point to aggressive 2-bit K-quants.
Sources: PRD.md:1-15, README.md:270-280, CLAUDE.md:10-25
Architecture and Data Flow
The quantization engine follows a modular design where oxidize-core owns the primitive tensor operations, and interface crates provide the user-facing conversion logic.
The diagram shows the standard requantization pipeline where source weights are processed into targeted quantized formats while undergoing quality validation.
Sources: oxidize-python/oxidize_python/quantize/cli.py:32-68, oxidize-c/include/oxidize/quant_analysis.h:32-45
Core Implementation
The Rust core defines the Quantization trait, which enforces a standard interface for all quantization types.
pub trait Quantization {
fn quantize(&self, data: &[f32], output: &mut [u8]) -> Result<()>;
fn dequantize(&self, data: &[u8], output: &mut [f32]) -> Result<()>;
fn block_size(&self) -> usize;
fn type_size(&self) -> usize;
}
Sources: PRD.md:86-93
Supported Quantization Types
oxidize supports various quantization types, each offering a different tradeoff between model size (compression) and output quality (perplexity).
| Type | Bits Per Element (Approx) | Description |
|---|---|---|
| F32 | 32.0 | Baseline 32-bit floating point. |
| F16 | 16.0 | 16-bit floating point; low-risk size reduction. |
| Q8_0 | 8.5 | 8-bit quantization; negligible quality loss. |
| Q6_K | 6.56 | 6-bit K-quant; near-lossless quality. |
| Q5_K_M | 5.5 | 5-bit K-quant (Medium); balanced speed/quality. |
| Q4_K_M | 4.5 | 4-bit K-quant (Medium); standard choice for most users. |
| Q3_K_M | 3.44 | 3-bit K-quant (Medium); strong memory savings. |
| Q2_K | 2.56 | 2-bit K-quant; maximum compression, high quality loss. |
Sources: oxidize-c/src/util/quant_analysis.c:204-230, oxidize-quantize/AGENTS.md:29-30
Quality Analysis and Metrics
To ensure model integrity, the project includes a quant_analysis module that measures the numerical divergence between original and quantized weights.
Key Metrics
- Mean Squared Error (MSE): Measures the average squared difference between original and dequantized weights.
- Cosine Similarity: Evaluates the directional similarity between weight vectors (1.0 indicates identity).
- Signal-to-Noise Ratio (SNR): Quantifies the ratio of signal strength to quantization noise in decibels (dB).
- Perplexity Delta: An estimate of how much the model's prediction error increases due to quantization.
Sources: oxidize-c/include/oxidize/quant_analysis.h:25-38, oxidize-c/src/util/quant_analysis.c:45-85
Recommendation Logic
The OcQuantRecommendation system suggests the optimal format based on user goals:
- OC_QUANT_GOAL_SPEED: Prioritizes types like Q8_0 or Q4_K for faster dequantization.
- OC_QUANT_GOAL_QUALITY: Recommends F16 or Q6_K to minimize perplexity increase.
- OC_QUANT_GOAL_MEMORY: Forces aggressive types like Q2_K to fit models in limited RAM.
Sources: oxidize-c/include/oxidize/quant_analysis.h:87-105, oxidize-c/src/util/quant_analysis.c:250-320
Tooling and Usage
oxidize-quantize CLI
The oxidize-quantize utility provides a standalone binary for converting model weights.
cargo run -p oxidize-quantize -- \
--input /path/to/input.bin \
--output /path/to/output.bin \
--source F32 \
--target Q4_0
Sources: README.md:148-154
Python and C Ports
The project maintains parallel implementations to ensure cross-platform compatibility:
- Python (
oxidize-python): Uses a_requantize_bodyloop to iterate through GGUF tensors, dequantizing from the source type and re-encoding to the target. - C (
oxidize-c): Provides a dependency-free C11 implementation using anOcArenafor memory management andmmapfor weight loading.
Sources: oxidize-python/oxidize_python/quantize/cli.py:32-58, CLAUDE.md:200-215
Summary
Model Quantization in oxidize enables high-performance inference by significantly reducing the memory footprint of LLMs. By providing both the conversion tools and the analysis framework to measure quality loss, the project allows developers to tune models for specific hardware constraints while maintaining predictable performance levels.
Sources: README.md:168-175, PRD.md:280-290