Weight Pruning
Relevant source files
The following files were used as context for generating this wiki page:
Weight pruning in the Oxidize project is the process of reducing model size and complexity by zeroing out or removing specific parameters from linear weight tensors. The project implements multiple pruning strategies, ranging from simple name-based filtering to advanced importance-based methods like Wanda and Magnitude pruning. These operations typically occur during model conversion or as a post-processing step on GGUF files to optimize them for local inference.
Sources: oxidize-prune/AGENTS.md:5-13, oxidize-c/include/oxidize/prune.h:3-8
Pruning Methodologies
Oxidize supports three primary pruning methods. The selection of a method depends on the desired speed, the availability of calibration data, and the required model quality.
Name-Filter Pruning
This method uses substring matching to keep or drop entire tensors based on their names. It is the fastest path because it copies bytes verbatim without performing weight-level calculations.
Sources: oxidize-prune/src/main.rs:20-23, oxidize-prune/AGENTS.md:7-9
Magnitude Pruning
Magnitude pruning identifies and zeros out parameters with the smallest absolute values (|W|). It calculates importance per output row, providing a structured approach to weight reduction without requiring external calibration data.
Sources: oxidize-c/src/util/prune.c:101-107, oxidize-prune/AGENTS.md:13-14
Wanda Pruning
Wanda (Weight and Activation) pruning calculates parameter importance using the formula |W_ij| · ‖X_j‖_2. In this formula, ‖X_j‖_2 represents the L2 norm of calibration activations for the input neuron. This method requires a pre-computed L2-norms cache from a calibration runner. It offers high quality, performing significantly faster than methods like SparseGPT while maintaining similar perplexity levels.
Sources: oxidize-prune/src/wanda.rs:18-20, oxidize-prune/AGENTS.md:10-12, oxidize-c/include/oxidize/prune.h:5-6
Architecture and Data Flow
The pruning system operates as a pipeline that processes GGUF files. The following diagram illustrates the flow from an unpruned input model to a pruned, re-quantized output.
The system identifies linear weights by checking if the tensor is 2-D, has an input dimension $\ge 64$, and contains .weight in its name (excluding embeddings or normalization layers).
Sources: oxidize-prune/src/wanda.rs:8-16, oxidize-prune/src/wanda.rs:202-214, oxidize-c/src/util/prune.c:134-138
Configuration and Options
Pruning behavior is controlled through various configuration structures and CLI arguments.
Pruning Options
The WandaOptions and OcPruneConfig structures define the parameters for a pruning run.
| Parameter | Type | Description |
|---|---|---|
sparsity | f32 | Fraction of weights to prune, in range [0, 1). |
pattern | SparsityPattern | Supports Unstructured, N2of4, and N4of8 patterns. |
joint_quantize | GgufQuantizationType | Optional target quantization for survivors (e.g., Q4_K_M). |
calibration | Path | Path to the activation L2-norms cache (required for Wanda). |
keep_names | Vec<String> | List of name substrings to exclude from pruning (default: token_embd, output, rope, norm). |
| Sources: oxidize-prune/src/wanda.rs:43-57.%20%7C%0A%7C%20%60pattern%60%20%7C%20%60SparsityPattern%60%20%7C%20Supports%20%60Unstructured%60%2C%20%60N2of4%60%2C%20and%20%60N4of8%60%20patterns.%20%7C%0A%7C%20%60joint_quantize%60%7C%20%60GgufQuantizationType%60%7C%20Optional%20target%20quantization%20for%20survivors%20(e.g.%2C%20Q4_K_M).%20%7C%0A%7C%20%60calibration%60%20%7C%20%60Path%60%20%7C%20Path%20to%20the%20activation%20L2-norms%20cache%20(required%20for%20Wanda).%20%7C%0A%7C%20%60keep_names%60%20%7C%20%60Vec%3CString%3E%60%20%7C%20List%20of%20name%20substrings%20to%20exclude%20from%20pruning%20(default%3A%20token_embd%2C%20output%2C%20rope%2C%20norm).%20%7C%0ASources%3A%20%5Boxidize-prune/src/wanda.rs%3A43-57), oxidize-c/include/oxidize/prune.h:20-29, oxidize-prune/src/main.rs:56-99 |
Sparsity Patterns
Oxidize implements structured sparsity patterns to optimize for specific hardware targets, such as NVIDIA's 2:4 structured sparsity.
- Unstructured: Any weight can be pruned.
- N2of4: Two weights are kept for every block of four.
- N4of8: Four weights are kept for every block of eight.
Sources: oxidize-prune/AGENTS.md:30, oxidize-prune/src/main.rs:114-118
Implementation Details
Calibration Cache Format
Wanda pruning relies on an L2-norms cache file. The file is text-based and contains one row per linear weight tensor. Each row includes the tensor name followed by N space-separated f32 values, corresponding to the input columns.
Sources: oxidize-prune/src/wanda.rs:252-257, oxidize-prune/AGENTS.md:43-47
Implementation in C
The C port mirrors the Rust implementation, providing standalone functions for magnitude and Wanda pruning. It uses OcGgufWriter to reconstruct a valid GGUF file from the pruned tensor data.
/* Find threshold at the sparsity quantile */
static float find_threshold(const float *data, size_t n, float sparsity) {
double sum_abs = 0.0;
for (size_t i = 0; i < n; i++) sum_abs += fabsf(data[i]);
float mean_abs = (float)(sum_abs / (double)n);
return mean_abs * sparsity;
}
Sources: oxidize-c/src/util/prune.c:86-94, oxidize-c/include/oxidize/prune.h:32-52
Integration with Model Conversion
The oxidize-convert tool allows pruning to be performed in a single pass during SafeTensors to GGUF conversion. When --prune is enabled, the converter generates an intermediate .prerun.gguf file, performs pruning and optional joint quantization, and then cleans up the intermediate artifact.
Sources: oxidize-convert/src/main.rs:94-117, oxidize-convert/AGENTS.md:23-28
Pruning provides a mechanism to reduce the memory footprint of large language models while maintaining as much performance as possible through importance-based weight selection. The inclusion of hardware-aligned patterns like N:M sparsity further optimizes the resulting models for modern compute backends.