Introduction to SeaStreamer

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/lib.rs
• sea-streamer-file/src/lib.rs
• sea-streamer-types/Cargo.toml
• sea-streamer-socket/Cargo.toml
• sea-streamer-redis/Cargo.toml
• sea-streamer-kafka/Cargo.toml
• sea-streamer-stdio/Cargo.toml

SeaStreamer is a comprehensive toolkit designed for building real-time stream processors in Rust. It emphasizes an asynchronous, non-blocking architecture that avoids locks on the "hot path," supporting both tokio and async-std runtimes to enable high concurrency. Sources: README.md:18-23, src/lib.rs:23-28

The system is designed to be backend-agnostic, providing a generic trait interface for different streaming services such as Redis and Kafka. This allows developers to write stream processing logic once and switch backends at runtime without recompiling the application. Sources: README.md:25-28, src/lib.rs:30-33

Core Architecture and Sub-crates

SeaStreamer is constructed as a modular ecosystem of sub-crates, each serving a specific role in the streaming pipeline. This modularity allows users to depend only on the components they need or use the top-level sea-streamer crate for a unified experience. Sources: README.md:104-114, src/lib.rs:105-115

Crate Hierarchy and Roles

Crate	Purpose	Key Features
sea-streamer-types	Abstract Interface	Defines core traits and shared data types without implementation.
sea-streamer-socket	Backend Selection	Concrete-type API for runtime backend switching (Kafka, Redis, etc.).
sea-streamer-kafka	Kafka Backend	Async wrapper for rdkafka and librdkafka.
sea-streamer-redis	Redis Backend	High-level async API for Redis Streams using message passing.
sea-streamer-stdio	Standard I/O	Real-time stream processing via Unix pipes and stdin/stdout.
sea-streamer-file	File Backend	Binary format (.ss) supporting real-time append and replay/seeking.
sea-streamer-runtime	Runtime Abstraction	Unified API for tokio and async-std.

Sources: README.md:116-160, Cargo.toml:34-40

System Dependency Flow

The following diagram illustrates how the sub-crates relate to the central sea-streamer toolkit.

The diagram shows the modular structure where sea-streamer-socket acts as the intermediary for concrete backend implementations. Sources: README.md:104-114, src/lib.rs:105-115

Backend Implementations

Redis Backend

The Redis implementation utilizes a mutex-free design, achieving concurrency through message passing. It abstracts low-level Redis commands like XADD and XREAD into high-level Producer and Consumer interfaces. It supports multiple modes including RealTime, Resumable, and LoadBalanced. Sources: README.md:183-205

Redis Concurrency Model

Separating read/write loops from the processing loop allows for parallel execution.

By separating these concerns, SeaStreamer ensures that the processor does not sit idle while waiting for I/O operations. Sources: README.md:214-222

Kafka Backend

The Kafka backend provides a typed system for KafkaConsumerOptions and KafkaProducer. It wraps the synchronous parts of librdkafka into an async API. Key capabilities include seeking to specific points in time, rewinding offsets, and manual commits. Sources: README.md:167-178

File Backend

sea-streamer-file is designed for both real-time streaming and historical replay. It uses a custom binary format (.ss) that avoids the overhead of delimiters by using a "Beacon" system. Sources: sea-streamer-file/src/lib.rs:10-24

• Beacons: Placed at fixed intervals (default 1MB), these act as in-place indexes for efficient fast-forwarding and seeking.
• Data Flow: Supports multi-producer and multi-consumer semantics, with load balancing performed via a round-robin algorithm.

Sources: sea-streamer-file/src/lib.rs:24-28, sea-streamer-file/src/lib.rs:59-60

Usage Patterns

Basic Consumer Setup

A SeaStreamer instance is created by connecting to a URI. Consumers are then spawned with specific SeaConsumerOptions. Sources: README.md:46-63

// Connect to a streamer (Redis, Kafka, or File)
let streamer = SeaStreamer::connect(stream.streamer(), Default::default()).await?;

// Configure consumer options
let mut options = SeaConsumerOptions::new(ConsumerMode::RealTime);
options.set_auto_stream_reset(SeaStreamReset::Earliest);

// Create the consumer for specific keys
let consumer: SeaConsumer = streamer
    .create_consumer(stream.stream_keys(), options)
    .await?;

Sources: README.md:46-59, src/lib.rs:45-58

Stream Processor Flow

The standard pattern for a processor involves a loop that consumes, transforms, and produces messages.

The send operation in SeaStreamer is non-blocking, allowing the process loop to continue immediately after handing off the message to the internal producer buffer. Sources: README.md:82-94

Summary of Features

1. Async Runtimes: Full support for tokio and async-std via sea-streamer-runtime. Sources: README.md:20-23
2. Generic Interface: Backend-agnostic traits allow swapping between local (File/Stdio) and cluster (Kafka/Redis) environments. Sources: README.md:25-28
3. Local Development: sea-streamer-stdio and sea-streamer-file enable testing without complex infrastructure. Sources: README.md:30-33
4. Performance: Implementation strategies like lock-free hot paths and separated I/O loops maximize throughput. Sources: README.md:20-21, README.md:214-220

SeaStreamer serves as a foundational layer for Rust developers to build low-resource, high-concurrency microservices for real-time data processing. Sources: README.md:35-37