High-Level Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/lib.rs
• sea-streamer-types/Cargo.toml
• sea-streamer-socket/Cargo.toml
• sea-streamer-runtime/Cargo.toml
• sea-streamer-file/src/lib.rs
• sea-streamer-stdio/Cargo.toml

SeaStreamer is a real-time stream processing toolkit for Rust designed to be async, generic, and testable. It provides a unified, backend-agnostic API that allows developers to write stream processing logic once and deploy it across various message streaming backends such as Kafka, Redis, or local files and standard I/O.

The system is built as a workspace of modular sub-crates, separating core traits and types from specific backend implementations and runtime abstractions. This modularity ensures that the toolkit remains lightweight while supporting high-performance, multi-threaded concurrency through message passing and async/non-blocking operations.

Sources: README.md:1-25, src/lib.rs:1-35

Component Hierarchy

The SeaStreamer ecosystem is organized into several layers, starting from core abstractions to concrete backend implementations.

Core Layers

• sea-streamer-types: Defines the foundational traits and types (e.g., Producers, Consumers, Messages) that all backends must implement.
• sea-streamer-socket: Provides a concrete, backend-agnostic API that allows switching between backends at runtime without recompilation.
• sea-streamer-runtime: An abstraction layer that aligns type signatures between different async runtimes like tokio and async-std.

Backend Implementations

• sea-streamer-kafka: Integration with Kafka and Redpanda using rdkafka.
• sea-streamer-redis: A high-level async API for Redis Streams featuring pipelined writes and paged reads.
• sea-streamer-stdio: Facilitates stream processing via Unix pipes and standard I/O for local development.
• sea-streamer-file: Enables persistent stream storage with support for real-time tailing and historical replay using a custom binary format (.ss).

Sources: README.md:154-177, src/lib.rs:168-185, sea-streamer-socket/Cargo.toml

Architecture Overview Diagram

The following diagram illustrates the relationship between the various crates and how they interact to provide a unified API.

The diagram shows the modular dependency flow where the application typically interacts with the socket layer, which abstracts the specific backend implementations and the underlying async runtime.
Sources: README.md:154-173, Cargo.toml:35-55

Backend Characteristics

Each backend implementation provides specific features tailored to its underlying technology while adhering to the common SeaStreamer traits.

Backend	Core Technology	Primary Use Case	Key Features
Kafka	librdkafka	Enterprise-grade streaming	Typed options, seek/rewind, async receipts, flush on disconnect.
Redis	Redis Streams	High-throughput messaging	Mutex-free, pipelined XADD, paged XREAD, load balancing.
File	Custom Binary (.ss)	Persistence & Replay	Beacon-based indexing, binary payload support, real-time + replay.
Stdio	Unix Pipes	Testing & CLI Tools	Line-based text messages, metadata support in [], pipeable.

Sources: README.md:197-320, sea-streamer-file/src/lib.rs:1-40

Stream Processing Logic Flow

In SeaStreamer, the read (consumer), process, and write (producer) loops are separated. This concurrency model, achieved via async message passing, ensures that the processor does not remain idle while waiting for I/O operations from the streaming server.

This diagram demonstrates the decoupled nature of data ingestion, processing, and output, allowing for maximum throughput by utilizing Rust's multi-threaded async environment.
Sources: README.md:255-275, src/lib.rs:104-125

Data Formats and Types

Message Structure

SeaStreamer uses a standardized message header across backends to ensure consistency when replaying or relaying data.

• Timestamp: ISO8601 or high-precision nanosecond timestamps.
• Stream Key: The topic or stream identifier.
• Sequence: The unique identifier for the message within the stream.
• Shard ID: The partition or shard identifier.

Stdio/File Format

For text-based backends like stdio, messages can include metadata in a bracketed prefix: [timestamp | stream_key | sequence | shard_id] payload. The file backend uses a more complex binary format with "Beacons" placed at fixed intervals (default 1MB) to allow for efficient seeking and alignment without message delimiters.

Sources: README.md:295-315, sea-streamer-file/src/lib.rs:25-30, sea-streamer-file/src/lib.rs:114-116

Runtime Abstraction

The sea-streamer-runtime crate allows the toolkit to be generic over the async executor. It provides common wrappers for:

• Task spawning and joining.
• Sleep/Timeout functions.
• File I/O (via tokio::fs or async-std::fs).
• Synchronization primitives like AsyncMutex.

Sources: README.md:343-346, sea-streamer-runtime/Cargo.toml:22-26, CHANGELOG.md:106-108

Conclusion

The high-level architecture of SeaStreamer emphasizes a "write once, stream anywhere" philosophy. By decoupling the core streaming logic from backend-specific implementations and runtime details, it provides a flexible environment for building low-resource, high-concurrency microservices in Rust. Its ability to transition from local file-based testing to distributed Kafka/Redis production environments without changing core logic is the central strength of its design.

Sources: README.md:15-35, src/lib.rs:40-60