Stream Joining & Fusion
Relevant source files
The following files were used as context for generating this wiki page:
Stream Joining and Fusion in the SeaStreamer ecosystem is managed by the sea-streamer-fuse crate. This module serves as a "stream processing toolbox," providing high-level utilities to manipulate and combine multiple data streams. Its primary purpose is to allow developers to merge disparate streams—potentially originating from different backends like Kafka, Redis, or local files—into a synchronized data flow.
The core functionality revolves around the StreamJoin component, which enables the joining of multiple streams based on message timestamps. This ensures that even if messages arrive out of order or at different rates from multiple sources, they can be processed in a temporally consistent manner.
Sources: sea-streamer-fuse/Cargo.toml:10, CHANGELOG.md:46-47
Architectural Role
sea-streamer-fuse is an extension crate that sits on top of the core sea-streamer-types and the backend-agnostic sea-streamer-socket. While the core crates handle connection management and message delivery, the Fuse module handles the logic of data transformation and multi-stream synchronization.
The following diagram illustrates the position of the Fuse module within the SeaStreamer crate hierarchy:
The sea-streamer-fuse crate depends on sea-streamer-types to utilize core message structures and Timestamp definitions.
Sources: README.md:143-152, sea-streamer-fuse/Cargo.toml:23, Cargo.toml:11
The StreamJoin Component
The StreamJoin component is the primary feature of the fusion system. It is designed to consume messages from multiple sources and output them as a single, unified stream.
Timestamp-Based Joining
Unlike simple concatenation, StreamJoin uses the message Timestamp (provided by sea-streamer-types) to align data. This is essential for applications like price feeds, telemetry merging, or audit logging where the relative timing of events across different streams is critical.
| Feature | Description |
|---|---|
| Temporal Alignment | Merges messages based on their UTC timestamps rather than arrival order. |
| Multi-Source Support | Can join streams from heterogeneous backends (e.g., merging a Redis stream with a Kafka stream). |
| Ordering Guarantees | Aims to provide a deterministic sequence of messages for downstream processors. |
Sources: CHANGELOG.md:46-47, sea-streamer-fuse/Cargo.toml:10
Stream Fusion Flow
The data flow within the fusion system involves multiple consumers feeding into a central joining logic, which then presents a unified interface to the stream processor.
This flow allows a SeaMessage from any backend to be processed alongside others, provided they implement the standard SeaStreamer traits.
Sources: README.md:162-180, CHANGELOG.md:46-47
Crate Dependencies & Configuration
The sea-streamer-fuse module is designed to be lightweight, relying on pinning and error handling utilities to manage asynchronous stream state without heavy overhead.
Key Dependencies
[dependencies]
thiserror = { version = "1", default-features = false }
pin-project = { version = "1.1" }
sea-streamer-types = { version = "0.5", path = "../sea-streamer-types" }
- thiserror: Used for defining custom error types related to fusion logic and stream state.
- pin-project: Necessary for the implementation of asynchronous stream polling and state management within the
StreamJoincomponent. - sea-streamer-types: Provides the fundamental
Message,Timestamp, andStreamKeytypes.
Sources: sea-streamer-fuse/Cargo.toml:18-23
Summary of Integration
The fusion system bridges the gap between raw message consumption and complex event processing. By utilizing the StreamJoin component, developers can treat multiple sources as a single temporal timeline, simplifying the logic required for multi-stream applications.
| Component | Role | Source |
|---|---|---|
sea-streamer-fuse | Main crate for stream processing tools. | sea-streamer-fuse/Cargo.toml |
StreamJoin | Logic for merging streams by timestamp. | CHANGELOG.md |
sea-streamer-types | Provides the underlying data models. | sea-streamer-types/Cargo.toml |
sea-streamer-socket | Provides the generic backend interface. | sea-streamer-socket/Cargo.toml |
The implementation is particularly useful in micro-service oriented architectures where data is distributed across various streaming platforms but needs to be unified for analytical or operational tasks.
Sources: README.md:25-30, CHANGELOG.md:46-47