Vision Processing & CLIP Encoder
Relevant source files
The following files were used as context for generating this wiki page:
The Vision Processing module provides CPU-side image preprocessing and a CLIP-style vision encoder. This system projects visual data into the language model's hidden space to support multimodal inference for models like LLaVA and Qwen-VL. It operates as a standalone CPU path without backend abstraction.
Sources: oxidize-core/src/vision/AGENTS.md, AGENTS.md
Architecture and Data Flow
The vision system transforms raw image bytes into embeddings that the language model interprets as tokens. The process involves resizing, normalization, and projection.
Vision Pipeline Flow
The diagram below shows the movement of data from a raw image file to the final multimodal prompt.
The pipeline begins with raw bytes and ends by interleaving visual embeddings with text tokens for the inference engine.
Sources: oxidize-core/src/vision/AGENTS.md, oxidize-c/include/oxidize/vision.h:58-75
Component Roles
| Component | Responsibility |
|---|---|
ImagePreprocessor | Resizes images using nearest-neighbor scaling and normalizes pixels. |
VisionEncoder | Loads weights and executes the forward pass to generate embeddings. |
MultimodalPrompt | Interleaves image embeddings at specific segment boundaries in text. |
VisionConfig | Defines geometry, patch size, and normalization presets. |
Sources: oxidize-core/src/vision/AGENTS.md, oxidize-c/include/oxidize/vision.h:35-56
CLIP-Style Encoder Implementation
The VisionEncoder implements a foundational CLIP path. It utilizes gemm_f32 for matrix multiplications and rms_norm_f32 for post-layer normalization. The current implementation performs patch embedding followed by projection. Note that the encoder does not yet include a transformer self-attention stack.
Sources: oxidize-core/src/vision/AGENTS.md
Encoder State and Weights
The encoder relies on specific weight matrices usually loaded from a GGUF model file. If weights are absent, the system uses mean-pooled patch statistics as a fallback.
| Weight Type | Description |
|---|---|
patch_proj | Linear projection for image patches. |
pos_emb | Positional embeddings for patches. |
ln_weight | Layer normalization weights. |
ln_bias | Layer normalization bias. |
cls_emb | Optional Class (CLS) token embedding. |
Sources: oxidize-c/include/oxidize/vision.h:46-56
Configuration and Presets
VisionConfig manages the geometry and token math for different vision models. Users select presets to match specific architectural requirements.
- clip_large: Uses 336px input, 14px patches, resulting in 576 tokens.
- clip_base: Standard CLIP configuration.
- llava_1_5: Configuration for LLaVA 1.5 multimodal models.
- qwen_vl: Preset for Qwen-VL models.
Sources: oxidize-core/src/vision/AGENTS.md, oxidize-c/include/oxidize/vision.h:35-43
Multimodal Prompting
The MultimodalPrompt structure facilitates the combination of text tokens and image embeddings. It inserts visual data at positions marked by a special image_token_id.
The sequence diagram illustrates how the MultimodalPrompt acts as a coordinator between the vision encoder and the LLM session.
Sources: oxidize-core/src/vision/AGENTS.md, oxidize-c/include/oxidize/vision.h:77-93
CLI Integration
The oxidize-cli supports vision tasks through specific command-line flags. When you enable vision mode, the CLI processes the provided image before starting the inference loop.
--vision: Enables multimodal mode.--image <PATH>: Specifies the path to the image for understanding.
Sources: oxidize-cli/src/main.rs:160-163
Summary
The Vision Processing & CLIP Encoder module enables the oxidize workspace to handle image data. It performs high-speed CPU preprocessing and projects visual features into a format compatible with LLM hidden states. This subsystem is critical for LLaVA-style multimodal workflows, allowing the engine to "see" and discuss visual inputs alongside text.
Sources: oxidize-core/src/vision/AGENTS.md, PRD.md