File Decoding Tools
Relevant source files
The following files were used as context for generating this wiki page:
File Decoding Tools in the SeaStreamer ecosystem are specialized utilities designed to read, parse, and visualize the .ss (sea-stream) binary file format. These tools allow developers to inspect message payloads, headers, and metadata stored within files produced by the sea-streamer-file backend. The suite includes both a Rust-based CLI utility (ss-decode) and a TypeScript implementation (sea-streamer-file-reader) specifically tailored for Node.js environments.
Sources: sea-streamer-file/README.md:23-25, sea-streamer-file/sea-streamer-file-reader/README.md:1-5
Rust Decoder (ss-decode)
The primary decoding tool is a binary utility called ss-decode, provided by the sea-streamer-file crate. It is designed to traverse .ss files, which utilize a custom message frame without delimiters to reduce encoding overhead. The decoder supports various output formats to make binary data human-readable.
Sources: sea-streamer-file/src/lib.rs:13-16, sea-streamer-file/src/lib.rs:25-30
Supported Output Formats
The ss-decode utility can transform binary stream data into the following textual representations:
| Format | Description | Example Output Component |
|---|---|---|
log | A human-readable format showing timestamps, stream keys, and sequences. | [2023-06-05T13:55:53.001 | hello | 1 | 0] message-1 |
ndjson | Newline Delimited JSON, structured for automated processing. | {"header":{"stream_key":"hello"...},"payload":"message-1"} |
Sources: sea-streamer-file/README.md:40-54
Rust Decoding Process Flow
The following diagram illustrates how the Rust decoder interacts with the .ss file format, utilizing Beacons for efficient seeking and alignment.
The SeaStreamerFile format uses Beacons at fixed intervals (default 1MB) as an in-place index to allow readers to align with message boundaries and facilitate seeking.
Sources: sea-streamer-file/src/lib.rs:18-22, sea-streamer-file/src/lib.rs:114
Node.js File Reader
The sea-streamer-file-reader is a TypeScript-based implementation that provides a lightweight, zero-dependency decoder for Node.js. Unlike the Rust implementation, it does not provide a high-level Consumer interface, focusing instead on simple message iteration.
Sources: sea-streamer-file-reader/README.md:1-9
Decoder Architecture (TypeScript)
The TypeScript decoder uses a MessageSource to iterate through the file. It distinguishes between UTF-8 text and binary blobs during visualization.
IPC and Subprocess Decoding
For complex reading tasks, the toolkit provides a subprocess implementation (subprocess.ts) that communicates via Inter-Process Communication (IPC). This allows for stateful operations like seeking to specific beacons without blocking the main process.
Control Commands (IPC):
open: Initializes theMessageSourcefor a specific file path.run: Begins streaming messages.seek: Rewinds the stream to the $n$-th beacon.more: Requests additional messages based on a read quota.
Sources: sea-streamer-file-reader/src/subprocess.ts:10-33, sea-streamer-file-reader/src/subprocess.ts:74-95
Key Technical Specifications
The decoding tools rely on the underlying SeaStreamer File format constants and structures.
| Constant/Component | Value/Description | Source |
|---|---|---|
DEFAULT_BEACON_INTERVAL | 1,048,576 bytes (1MB) | src/lib.rs:114 |
StreamMode | Live, Replay, or LiveReplay | subprocess.ts:145 |
SEA_STREAMER_WILDCARD | Special key to subscribe to all streams in a file | CHANGELOG.md:27 |
READ_QUOTA | 10,000 (Subprocess message limit) | subprocess.ts:8 |
Beacon Management
Beacons act as summaries of the streams and allow for alignment. In the Node.js decoder, the source tracks the current beacon index. If the beacon changes during reading, the decoder logs the updated index metadata.
Sources: sea-streamer-file-reader/src/decoder.ts:37-41, sea-streamer-file/src/lib.rs:18-21
Conclusion
The File Decoding Tools are essential for managing and debugging SeaStreamer's binary file backend. By providing both a high-performance Rust utility for CLI usage and a TypeScript implementation for Node.js integration, the project ensures that .ss files remain accessible across different runtime environments. These tools leverage the format's Beacon system to provide efficient navigation and human-readable visualization of complex binary streams.