Video Processing & Temporal Fusion
Relevant source files
The following files were used as context for generating this wiki page:
Video processing within the Oxidize workspace provides a multimodal path for handling video data. It samples frames from a video source, encodes individual frames, and aggregates them temporally to create embeddings suitable for language model consumption. This module serves as the video-specific analogue to the Vision module, extending multimodal capabilities from static images to time-series visual data.
The system relies on external tools like ffmpeg for video decoding and integrates with the core inference engine to interleave video features with text prompts. Temporal fusion is the final step in this pipeline, where individual frame embeddings are merged into a unified representation that captures the temporal dynamics of the video.
Sources: oxidize-core/src/video/AGENTS.md, CLAUDE.md:73, oxidize-core/src/vision/AGENTS.md:37
Architecture and Pipeline
The video processing pipeline follows a linear data flow from raw video input to multimodal prompt assembly. The module is structured into several specialized components that handle decoding, sampling, encoding, and fusion.
Data Flow Diagram
The following diagram illustrates the flow of data through the video processing system:
Module Structure
The video processing logic is organized into several key files within oxidize-core/src/video/:
| Component | File | Responsibility |
|---|---|---|
| Decoder | decoder.rs | Interfaces with ffmpeg to decode video streams into raw frames. |
| Sampler | frame_sampler.rs | Determines which frames to extract based on timestamps and sampling strategies. |
| Encoder | encoder.rs | Projects individual frames into the model's hidden space. |
| Temporal | temporal.rs | Fuses multiple frame embeddings into a single temporal representation. |
| Prompt | prompt.rs | Assembles the final prompt by interleaving video and text segments. |
| Config | config.rs | Defines VideoConfig, TemporalConfig, and sampling strategies. |
Sources: oxidize-core/src/video/AGENTS.md:11-21, oxidize-core/src/lib.rs:104-106
Frame Sampling and Encoding
Video processing begins by sampling specific frames from the input stream. This is necessary to manage context window limits and computational overhead, as processing every frame of a high-framerate video is often redundant for LLM understanding.
Sampling Strategies
The FrameSamplingStrategy defined in config.rs dictates how the system selects frames. It uses timestamps to identify relevant points in the video for extraction.
Sources: oxidize-core/src/video/AGENTS.md:12, 24
Per-Frame Encoding
Once frames are sampled and preprocessed, the VisionEncoder (reused or adapted from the vision module) encodes each frame. The encoder performs patch embedding and projects the visual data into the hidden dimension used by the LLM.
Sources: oxidize-core/src/video/AGENTS.md:25, oxidize-core/src/vision/AGENTS.md:15
Temporal Fusion
Temporal fusion aggregates individual frame embeddings across the time dimension. This process is essential for the model to understand motion, sequence, and duration within the video.
Temporal Aggregation Flow
The aggregation logic resides in temporal.rs and utilizes the TemporalPool configuration.
Aggregation Logic
The system supports different methods for aggregation, defined by the TemporalPool enum. These methods consolidate the spatial-temporal patches into a format the inference engine can process as a multimodal sequence.
Sources: oxidize-core/src/video/AGENTS.md:12, 26
Multimodal Prompt Assembly
The final stage of the video path is the assembly of a MultimodalPrompt. This component interleaves the fused video embeddings with text segments.
- Segment Boundaries: Video embeddings are injected at specific segment boundaries within the prompt.
- Token Interleaving: The process mirrors the
visionmodule'sMultimodalPrompt, which manages the transition between visual tokens and text tokens. - Interface Integration: The assembled prompt is passed to the inference engine via
oxidize-coremodel traits.
Sources: oxidize-core/src/video/AGENTS.md:27, oxidize-core/src/vision/AGENTS.md:22, CLAUDE.md:73
Configuration and Subsystems
The video module is controlled through several configuration structures:
| Structure | Location | Description |
|---|---|---|
VideoConfig | config.rs | General settings for video processing (resolution, framerate). |
TemporalConfig | config.rs | Defines how frames are fused and the aggregation window. |
FrameSamplingStrategy | config.rs | Enum defining uniform, keyframe-based, or random sampling. |
TemporalPool | config.rs | Specifies the mathematical operation for fusion (mean, max, or attention-based). |
Sources: oxidize-core/src/video/AGENTS.md:12
Summary
Video Processing & Temporal Fusion in Oxidize enables the engine to interpret video data by transforming visual streams into temporal embeddings. The module orchestrates a pipeline of decoding, sampling, and encoding, culminating in temporal fusion that captures the relationship between frames. This multimodal representation is then seamlessly integrated with text prompts for high-level inference tasks.
Sources: oxidize-core/src/video/AGENTS.md, oxidize-core/src/lib.rs:104-106