Checkpoint Merging
Relevant source files
The following files were used as context for generating this wiki page:
Checkpoint Merging is a system within the oxidize workspace that combines two HuggingFace SafeTensors or GGUF checkpoints into a single output model. The system uses mathematical blending techniques like linear interpolation and Spherical Linear Interpolation (SLERP) to merge model weights. It supports per-tensor category weighting, allowing fine-grained control over how attention and MLP layers are blended.
Sources: oxidize-merge/AGENTS.md:1-6, CLAUDE.md:65-65
Architecture and Components
The oxidize-merge crate provides both a library API and a Command Line Interface (CLI) for blending checkpoints. It handles sharded input and output, supporting large models by breaking them into manageable parts.
The diagram shows the flow from user input through recipe construction to the core merging logic and final sharded output.
Sources: oxidize-merge/AGENTS.md:9-19, oxidize-merge/src/main.rs:104-124
Core Components
| Component | Description |
|---|---|
MergeRecipe | Defines the specific weights (t) for different tensor categories (Attention, MLP, etc.). |
MergeOptions | Configures the paths for models A and B, output path, blend method, and missing tensor policies. |
Shard Index | Manages HuggingFace model.safetensors.index.json for reading sharded models. |
Sharded Writer | Writes merged tensors to disk, defaulting to a maximum shard size of 5 GiB. |
Sources: oxidize-merge/AGENTS.md:22-29, oxidize-merge/src/main.rs:8-10
Blending Methods
The system supports multiple mathematical strategies for merging weights between two models.
- Linear: A weighted linear combination of tensors.
- SLERP: Spherical Linear Interpolation, which preserves the geometric characteristics of the weight space.
- TIES/DARE: Advanced strategies supported in the C port for trimming and rescaling parameters.
Sources: oxidize-merge/src/main.rs:72-76, oxidize-c/include/oxidize/merge.h:23-28
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
--method | Enum | slerp | The blend algorithm to use (linear or slerp). |
--t | Float | None | Global blend weight in [0, 1] toward model B. |
--attention-t | Float | 0.3 | Weight for attention tensors toward model B. |
--mlp-t | Float | 0.5 | Weight for MLP/Expert tensors toward model B. |
--other-t | Float | 0.4 | Weight for all other float tensors. |
--missing | Enum | error | Policy for tensors only in one model (error, a, b). |
Sources: oxidize-merge/src/main.rs:12-62, oxidize-merge/AGENTS.md:32-37
Implementation Details
The merge_models function is the primary entry point. It processes tensors according to a MergeRecipe. If a global t value is provided, it overrides all category-specific presets.
The sequence shows how user arguments are converted into a recipe before the core processing logic generates the final report.
Sources: oxidize-merge/src/main.rs:104-142, oxidize-merge/src/blend.rs
Presets
The system includes predefined recipes for specific models, such as the KimiK275 preset. This allows users to apply optimized weights for known model architectures without manual tuning.
Sources: oxidize-merge/src/main.rs:78-81, oxidize-merge/src/main.rs:152-154
Summary
Checkpoint Merging enables the creation of hybrid models by blending the weights of two distinct checkpoints. By providing flexible weighting for different tensor categories and supporting both standard linear and spherical interpolation, it allows developers to fine-tune the resulting model's behavior. The inclusion of sharding and missing-tensor policies ensures the system is robust enough for production-scale LLM architectures.