Standard I/O (Stdio) Backend
Relevant source files
The following files were used as context for generating this wiki page:
The Standard I/O (Stdio) backend for SeaStreamer provides a real-time stream processing interface using Unix pipes (stdin/stdout). It is designed to enable the connection of stream processors via standard shell piping (e.g., processor_a | processor_b), facilitating local development, testing, and modular tool chains without the need for a dedicated streaming cluster like Kafka or Redis.
Sources: README.md:214-216, sea-streamer-stdio/src/lib.rs:1-3
Unlike the File Backend, the Stdio backend operates exclusively in real-time and handles only UTF-8 text data. It supports multi-producer and multi-consumer semantics within a single process, utilizing a global streamer instance to coordinate message distribution.
Sources: sea-streamer-file/src/lib.rs:7-11, README.md:237-238
System Architecture and Data Flow
The Stdio backend translates lines of text from standard input into SeaMessage objects and serializes outgoing messages to standard output. A single StdioStreamer exists per process, managing the lifecycle of producers and consumers.
Message Processing Flow
When data is received via stdin, the backend attempts to parse each line. If a line matches a specific metadata format, it extracts the timestamp, stream key, sequence number, and shard ID. Otherwise, it treats the entire line as a raw string payload and assigns it to a default broadcast stream key.
The diagram above illustrates how input is parsed and dispatched to consumers. Consumers within the same group benefit from round-robin load balancing.
Sources: README.md:231-235, README.md:240-243
Message Format
The Stdio backend supports both raw text and a structured format for including message metadata.
Supported Metadata Syntax
Messages can be prefixed with metadata enclosed in square brackets:
[timestamp | stream_key | sequence | shard_id] payload
| Field | Description | Requirement |
|---|---|---|
timestamp | ISO8601 formatted string | Optional |
stream_key | The identifier for the stream | Optional (Defaults to broadcast) |
sequence | Incremental message sequence number | Optional |
shard_id | Identifier for stream partitioning | Optional |
Sources: README.md:218-228, sea-streamer-stdio/src/lib.rs:16-26
Example Valid Messages
[2022-01-01T00:00:00.123 | my_topic] "a string payload"
[my_topic | 123 | 4] { "payload": "anything" }
a plain, raw message
Sources: README.md:220-227
Components and Features
Streamer, Producers, and Consumers
The Stdio backend implements the standard SeaStreamer traits:
- StdioStreamer: The central hub for connecting to the stdio "socket".
- StdioProducer: Handles writing messages to
stdout. - StdioConsumer: Handles reading and filtering messages from
stdin.
Load Balancing
Consumers belonging to the same ConsumerGroup receive messages in a round-robin fashion. This allows developers to spawn multiple async tasks to process a single input stream in parallel.
Sources: README.md:240-243, sea-streamer-stdio/src/lib.rs:36-39
Technical Constraints
| Feature | Stdio Backend Support |
|---|---|
| Data Type | UTF-8 String only |
| Persistence | None (Real-time only) |
| Replay/Seek | Not supported (See File Backend) |
| Instances | One Streamer per process |
Sources: sea-streamer-file/src/lib.rs:7-11, sea-streamer-stdio/src/lib.rs:1-8
Implementation Details
The backend utilizes nom for parsing the message metadata format and flume for internal async message passing between the input reader task and the consumers.
The sequence diagram shows the flow of messages through the Streamer to the system I/O handles.
Sources: sea-streamer-stdio/Cargo.toml:19-21, sea-streamer-stdio/src/lib.rs:30-45
Usage in CLI
The Stdio backend is often used with the relay utility to bridge local testing with production clusters:
# Stream from Stdio to Redis
clock | relay --input stdio:///clock --output redis://localhost:6379/clock
Sources: sea-streamer-socket/src/lib.rs:25-28
Conclusion
The Standard I/O Backend is a lightweight, zero-configuration streaming implementation for the SeaStreamer ecosystem. By leveraging Unix pipes and a simple text-based protocol, it enables highly portable and testable stream processing logic that can be seamlessly transitioned to Kafka or Redis in production environments.