File Backend Implementation
Relevant source files
The following files were used as context for generating this wiki page:
- sea-streamer-file/src/lib.rs
- sea-streamer-file/src/format.rs (Referenced via tests and module declarations)
- README.md
- sea-streamer-file/README.md
- sea-streamer-file/tests/loopback.rs
- sea-streamer-file/sea-streamer-file-reader/README.md
The sea-streamer-file backend provides a high-performance, async implementation for stream processing using local files as the storage medium. Unlike the stdio backend which is limited to real-time UTF-8 data, the File backend supports both real-time streaming and historical replay of binary data. It utilizes a custom .ss (sea-stream) file format designed for efficient seeking and fast-forwarding.
Sources: sea-streamer-file/src/lib.rs:3-10, README.md:278-285
File Format and Architecture
The SeaStreamer File format is designed to eliminate the overhead of encoding/decoding message delimiters. To facilitate efficient navigation through large binary files, the implementation embeds "Beacons" at fixed intervals. These beacons act as an in-place index, containing stream summaries that allow readers to align with message boundaries and seek to specific timestamps or offsets without scanning the entire file.
Sources: sea-streamer-file/src/lib.rs:16-21, sea-streamer-file/README.md:14-17
Component Overview
The backend is composed of several specialized modules handling different aspects of the file lifecycle:
| Component | Responsibility |
|---|---|
FileSink | Handles low-level writing of bytes and markers to the physical file. |
FileSource | Manages reading, seeking, and tracking the current position within a file. |
MessageSink | High-level producer interface that manages message framing and beacon insertion. |
MessageSource | High-level consumer interface that decodes frames and utilizes beacons for seeking. |
Beacon | An in-place index structure containing stream metadata and running checksums. |
Sources: sea-streamer-file/src/lib.rs:59-71, sea-streamer-file/tests/loopback.rs:245-255
Data Flow Diagram
The following diagram illustrates how messages move from a producer through the File Sink into the .ss file, and how a consumer retrieves them via the File Source.
Core Data Structures
Beacon Mechanism
Beacons are inserted every DEFAULT_BEACON_INTERVAL (1MB by default). They contain remaining_messages_bytes to handle messages that span across beacon boundaries and a list of Marker items summarizing stream state.
pub struct Beacon {
pub remaining_messages_bytes: u32,
pub items: Vec<Marker>,
}
pub struct Marker {
pub header: MessageHeader,
pub running_checksum: Checksum,
}
Sources: sea-streamer-file/src/lib.rs:78, sea-streamer-file/tests/loopback.rs:104-118
Message Framing
Messages are stored as a combination of a fixed-size header and a variable-length payload. The header includes the StreamKey, ShardId, Sequence, and Timestamp.
Sources: sea-streamer-file/tests/loopback.rs:71-85
Seeking and Replay Logic
The File backend supports multiple ReadFrom modes, allowing consumers to start from the beginning of a file or seek to a specific position.
Seeking Sequence
When a seek request is issued (by sequence number or timestamp), the system uses the internal beacon index to jump to the nearest preceding synchronization point before fine-tuning the position by reading individual messages.
Configuration Defaults
The implementation defines several constants that govern file behavior:
| Constant | Value | Description |
|---|---|---|
DEFAULT_BEACON_INTERVAL | 1,048,576 bytes | Interval between in-place index markers. |
DEFAULT_FILE_SIZE_LIMIT | 16 GB | Maximum size a single stream file can reach. |
DEFAULT_PREFETCH_MESSAGE | 1000 | Number of messages to buffer ahead. |
Sources: sea-streamer-file/src/lib.rs:78-80
Implementation Details: Write and Sync
The FileSink and MessageSink work together to ensure data integrity through checksums and flushes.
- Write: Messages are written to a buffer.
- Checksum: A CRC checksum is calculated for each message and updated in the running stream state.
- Beacon Check: If the beacon interval is reached, a
Beaconframe is injected. - Flush: The
flush()call ensures data is persisted to the underlying storage runtime (Tokio or async-std).
Sources: sea-streamer-file/tests/loopback.rs:40-60, sea-streamer-file/src/lib.rs:25-30
Conclusion
The File Backend implementation in SeaStreamer provides a robust mechanism for persistent stream storage. By using a custom binary format with embedded beacons, it achieves a balance between high-throughput sequential writes and low-latency random access for replaying historical data. This makes it an essential component for local development, testing, and applications requiring durable event logging like FireDBG.
Sources: README.md:278-285, sea-streamer-file/src/lib.rs:5-15