Message Data Model
Relevant source files
The following files were used as context for generating this wiki page:
The Message Data Model in SeaStreamer defines the core structure of data packets transmitted across the toolkit's various backends (Kafka, Redis, File, and Stdio). It establishes a unified abstraction for stream data, consisting of a structured header for routing and ordering, and a flexible payload for application-specific data.
This model is designed to be backend-agnostic, allowing developers to write stream processing logic that works seamlessly across different streaming infrastructures. In the context of the File backend, this model is further extended with checksums and framing logic to ensure data integrity during persistence and replay.
Core Message Structure
The fundamental unit of data in SeaStreamer is represented by the Message trait and its concrete implementations like OwnedMessage and SharedMessage. Every message is composed of two primary parts: the Header and the Payload.
Message Header
The header contains metadata essential for stream management, identifying the source, position, and timing of the data.
| Field | Type | Description |
|---|---|---|
stream_key | StreamKey | Identifies the stream or topic the message belongs to. |
shard_id | ShardId | Identifies the specific partition or shard within the stream. |
sequence | SeqNo | A unique, usually incrementing identifier for the message within its shard. |
timestamp | Timestamp | The time the message was produced or recorded. |
Sources: sea-streamer-types/src/message.rs:85-98, sea-streamer-file/src/format.rs:188-196
Message Payload
The payload represents the actual data being streamed. In the core library, this is abstracted as a Buffer, which can be converted to bytes or UTF-8 strings.
The diagram shows the relationship between the base Message interface and the concrete implementations used for ownership-based or shared-access processing.
Sources: sea-streamer-types/src/message.rs:20-65
File Backend Representation
When messages are persisted using sea-streamer-file, they are wrapped in a custom binary frame. This frame includes the message header, the payload size, the payload itself, a CRC16 checksum, and a delimiter for semi-human readability.
Binary Message Frame
The physical layout of a message in a .ss file is as follows:
- Header: Serialized
MessageHeader(StreamKey length + StreamKey + ShardId + SeqNo + Timestamp). - Size: A 32-bit big-endian integer representing the payload length.
- Payload: The raw binary data.
- Checksum: A 16-bit CRC16/CDMA2000 checksum of the payload.
- Delimiter: A single byte
0x0D(\r) to prevent issues with text editors.
Sources: sea-streamer-file/src/format.rs:25-30, sea-streamer-file/src/format.rs:141-155
Data Integrity with Checksums
The sea-streamer-file backend uses crc16_cdma2000 to validate message integrity. Each message stored includes a checksum of its payload. Additionally, "Beacons" are placed at fixed intervals (default 1MB) containing "Markers" with running checksums to facilitate efficient seeking and verification.
This flowchart illustrates the sequential process of serializing a message into the SeaStreamer binary file format.
Sources: sea-streamer-file/src/format.rs:157-167, sea-streamer-file/src/messages.rs:370-385
Specialized Message Types
SeaStreamer reserves a specific stream key, SEA_STREAMER_INTERNAL, for control signals within the data model.
- End-of-Stream (EOS): A message with the
EOSpayload. It signals the graceful termination of a stream, allowing consumers to stop processing. - Pulse: An internal message used to keep stream connections alive or signal activity without carrying application data.
Sources: sea-streamer-file/src/messages.rs:21-22, sea-streamer-file/src/messages.rs:480-496
Summary of Data Components
| Component | Description | Relevance |
|---|---|---|
Beacon | Summary of streams at fixed file intervals. | Seeking & Indexing |
Marker | An entry in a Beacon containing a MessageHeader and Checksum. | Replay Alignment |
RunningChecksum | A cumulative CRC16 value updated with every message. | Stream Validation |
ShortString | A string prefixed by a 1-byte length (max 255 chars). | StreamKey Encoding |
Sources: sea-streamer-file/src/format.rs:42-55, sea-streamer-file/src/format.rs:251-265
The SeaStreamer Message Data Model provides a robust foundation for stream processing by strictly separating routing metadata from the data payload, while providing the necessary hooks for backends to implement durability and integrity features like the binary framing and checksumming found in the File backend.