Model Conversion Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Model Conversion Pipeline transforms high-precision frontier models (e.g., FP8, BF16) into compressed int4 or int8 containers optimized for the Colibrì inference engine. This pipeline ensures that massive Mixture-of-Experts (MoE) models can run on consumer hardware by reducing their memory footprint while maintaining token-exact output accuracy against reference implementations.
The process includes downloading model shards, mapping internal tensor names to the project's expected format, and applying row-wise symmetric quantization. The pipeline handles multi-gigabyte shards individually to minimize peak disk and RAM usage.
Sources: README.md:126-135, c/tools/convert_olmoe_merged.py:5-15
Architecture and Data Flow
The pipeline operates in a streaming or local mode to process model weights. It iterates through safetensors shards, applies transformations to specific tensor types (like routed experts), and writes optimized output shards.
The diagram shows the transition from high-precision source weights to optimized Colibrì containers through selective quantization and name mapping.
Sources: c/tools/convert_olmoe_merged.py:102-140, c/tools/convert_inkling_int4.py:84-118
Core Components
1. Download and Acquisition
The pipeline acquires source weights from ModelScope or HuggingFace. It uses parallel downloads and supports resuming interrupted transfers.
- Parallelism: Defaults to 3-4 parallel shards.
- Verification: Checks file sizes and safetensors headers for integrity.
Sources: c/download_fp8.py:5-20, README.md:204-210
2. Quantization Logic
The pipeline applies row-wise symmetric quantization to routed experts, which represent the majority of model parameters.
- int8 Quantization: Scales weights based on the maximum absolute value per row, clipping values to the -128 to 127 range.
- int4 Quantization: Packs weights into 4-bit nibbles (low nibble for even columns, high nibble for odd columns) with f32 row scales.
Sources: c/tools/convert_inkling_int4.py:34-54, c/tools/convert_olmoe_merged.py:41-47
3. Tensor Mapping and Merging
Different model architectures require specific name mappings to align with the colibri engine's expectations.
- Name Mapping: Renames native tensor names (e.g.,
model.llm.embed.weight) to standardized tokens (e.g.,model.embed_tokens.weight). - Expert Merging: Consolidates
gate_proj,up_proj, anddown_projinto a singlemerged_weighttensor per expert to enable single-read disk access during inference. - De-interleaving: Handles fused tensors (like w13 gate/up projections) by de-interleaving rows to group gate and up weights.
Sources: c/tools/convert_olmoe_merged.py:5-9, c/tools/convert_inkling_int4.py:57-81
Key Functions and Classes
| Component | Description |
|---|---|
quant_int4_rows | Packs f32 weights into u8 nibbles with f32 scales. |
map_name | Regulates tensor naming conventions via regex substitution. |
OutputWriter | Buffers converted tensors and flushes them into new .safetensors shards. |
convert_expert | Quantizes and concatenates expert projections (gate/up/down) into one tensor. |
download_file_ms | Downloads shards from ModelScope CDN with fallback support. |
Sources: c/tools/convert_inkling_int4.py:44-54, c/tools/convert_olmoe_merged.py:65-88, c/download_fp8.py:47-54
Conversion Workflow
The sequence demonstrates the "one shard at a time" strategy used to bound RAM and disk usage during conversion.
Sources: c/tools/convert_olmoe_merged.py:165-200, c/tools/convert_inkling_int4.py:151-175
Conclusion
The Model Conversion Pipeline is a critical utility that enables large-scale MoE models to operate within the memory constraints of consumer hardware. By applying specialized quantization and merging experts into unified structures, it bridges the gap between datacenter-class weights and local, C-based inference.
Sources: README.md:65-75, c/tools/README.md