SeaStreamer File Format (.ss)
Relevant source files
The following files were used as context for generating this wiki page:
The SeaStreamer File Format (.ss) is a specialized binary format designed for high-performance stream processing, supporting both real-time appending and historical replay. Unlike the Stdio backend which is limited to UTF-8 text, the File backend handles arbitrary binary data and allows multiple independent streamers within a single process. It is a core component of the sea-streamer-file crate and powers the event stream of FireDBG.
The format is characterized by its lack of message delimiters, which eliminates encoding/decoding overhead for payloads but requires a structured frame. Efficiency in fast-forwarding and seeking is achieved through an array of "Beacons" placed at fixed intervals (typically every 1MB). These beacons act as an in-place index, allowing readers to align with message boundaries and providing summaries of the streams contained within the file.
Sources: sea-streamer-file/src/lib.rs:8-23, sea-streamer-file/README.md:8-18, README.md:200-205
Architecture and Design Principles
The .ss format is designed to mimic the behavior of a tail -f command while adding custom message framing for binary payloads. It supports multi-producer and multi-consumer semantics, with load balancing handled in a round-robin fashion similar to other SeaStreamer backends.
Structural Components
The file consists of three primary high-level structures:
- File Header: Contains metadata such as the file name, creation timestamp, and beacon interval.
- Message Frames: Custom frames carrying binary payloads without the use of delimiters.
- Beacons: Summary blocks placed at fixed byte intervals to facilitate seeking.
The diagram shows the sequential nature of the .ss format, where Beacons interleave with message frames at fixed intervals.
Sources: sea-streamer-file/src/lib.rs:16-27, sea-streamer-file-reader/README.md:16-17
Message Framing
SeaStreamer File does not use delimiters (like newlines) to separate messages. Instead, it uses a custom message frame. This design choice removes the overhead of encoding/decoding message payloads but necessitates a header-driven approach to determine message boundaries.
Message Header Attributes
Each message includes a header that contains routing and sequence information:
- Timestamp: The time the message was produced.
- Stream Key: The identifier for the specific stream (topic).
- Sequence: A unique identifier for the message within its stream.
- Shard ID: The identifier for the shard (currently defaults to Shard ZERO).
Sources: sea-streamer-file/src/lib.rs:40-45, sea-streamer-file/README.md:38-43, sea-streamer-file-reader/src/decoder.ts:25-28
Indexing and Seeking via Beacons
Beacons are the primary mechanism for efficient file navigation. They allow the system to seek or rewind to specific timestamps or offsets without scanning every individual message frame.
Beacon Functionality
- Alignment: Beacons allow readers to re-align with message boundaries if they jump to an arbitrary offset.
- Indexing: Each beacon contains a summary of the streams, acting as an in-place index.
- Interval: The default beacon interval is 1,048,576 bytes (1MB).
Seeking workflow utilizing Beacons to avoid full file scans.
Sources: sea-streamer-file/src/lib.rs:20-23, sea-streamer-file/src/lib.rs:85
Configuration and Limits
The file backend provides several default constants and configuration options to manage file growth and performance.
| Parameter | Default Value | Description |
|---|---|---|
DEFAULT_BEACON_INTERVAL | 1,048,576 (1MB) | The byte interval at which Beacons are inserted. |
DEFAULT_FILE_SIZE_LIMIT | 17,179,869,184 (16GB) | The default maximum size for a single .ss file. |
DEFAULT_PREFETCH_MESSAGE | 1,000 | Number of messages to pre-load during reading. |
SEA_STREAMER_WILDCARD | "SEA_STREAMER_WILDCARD" | Reserved key to subscribe to all streams in a file. |
Sources: sea-streamer-file/src/lib.rs:85-92, CHANGELOG.md:19-20
Implementation Details
Crate Dependencies
The sea-streamer-file backend relies on several key libraries for its functionality:
- flume: Used for async message passing.
- notify: Used for watching file changes in real-time.
- fastrand: Used for internal random operations.
- sea-streamer-runtime: Provides the async abstraction (Tokio or async-std).
Sources: sea-streamer-file/Cargo.toml:23-29
Decoder Utilities
A utility program ss-decode is provided to inspect .ss files. It supports multiple output formats:
- log: A human-readable text format showing headers in square brackets.
- ndjson: Newline-delimited JSON for machine processing.
// Example of how a message is processed in the TypeScript reader
const h = message.header;
console.log(
`[${h.timestamp.toISOString()} | ${h.streamKey.name} | ${h.sequence.no} | ${h.shardId.id}]`,
isUtf8(message.payload.buffer) ? message.payload.toString() : "<BINARY BLOB>"
);
Sources: sea-streamer-file/src/lib.rs:32-38, sea-streamer-file-reader/src/decoder.ts:24-28
Summary
The SeaStreamer File Format (.ss) provides a robust solution for local stream storage and replay. By utilizing a beacon-based indexing system and raw binary framing, it achieves a balance between the simplicity of file-based logging and the performance requirements of modern stream processing. While currently limited to single-shard operations (Shard ZERO) and lacking built-in resumable state management, its design enables high-throughput data ingestion and precise temporal seeking.
Sources: sea-streamer-file/src/lib.rs:77-81, README.md:200-205