Quick Start Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/lib.rs
• Cargo.toml
• sea-streamer-file/src/lib.rs
• sea-streamer-socket/Cargo.toml
• sea-streamer-types/Cargo.toml

SeaStreamer is a comprehensive toolkit designed for building real-time stream processors in Rust. It provides an asynchronous, non-blocking API that remains backend-agnostic through a generic trait interface, supporting various backends such as Kafka, Redis, Stdio, and File.

This guide provides the necessary steps to integrate SeaStreamer into your project, configure basic producers and consumers, and execute stream processing tasks across different supported backends.

Sources: README.md:1-20, src/lib.rs:1-25

Installation and Configuration

To begin using SeaStreamer, add the crate to your Cargo.toml with the desired backend features. SeaStreamer supports both tokio and async-std runtimes.

[dependencies]
sea-streamer = { version = "0.5", features = ["kafka", "redis", "stdio", "socket", "runtime-tokio"] }

Supported Backend Features

Feature	Backend Description
kafka	Integration with Kafka / Redpanda via rdkafka.
redis	Integration with Redis Streams.
stdio	Standard I/O for pipe-based stream processing.
file	Binary file-based streaming with replay and seeking capabilities.
socket	Backend-agnostic API for runtime backend selection.
runtime-tokio	Enables support for the Tokio async runtime.
runtime-async-std	Enables support for the async-std runtime.

Sources: Cargo.toml:35-46, README.md:27-31, src/lib.rs:32-34

Basic Stream Operations

SeaStreamer utilizes a unified API for connecting to streamers, creating producers, and consuming messages.

Connecting and Consuming

To consume messages, you establish a connection using a StreamerUri and create a SeaConsumer.

let streamer = SeaStreamer::connect(stream_uri, Default::default()).await?;
let mut options = SeaConsumerOptions::new(ConsumerMode::RealTime);
options.set_auto_stream_reset(SeaStreamReset::Earliest);

let consumer: SeaConsumer = streamer
    .create_consumer(stream_keys, options)
    .await?;

loop {
    let mess: SeaMessage = consumer.next().await?;
    println!("[{}] {}", mess.timestamp(), mess.message().as_str()?);
}

Sources: README.md:37-56, src/lib.rs:40-59

Producing Messages

Producers send messages to specific stream keys. The send operation is non-blocking.

let streamer = SeaStreamer::connect(stream_uri, Default::default()).await?;
let producer: SeaProducer = streamer
    .create_producer(stream_key, Default::default())
    .await?;

producer.send(format!("message payload"))?;
producer.end().await?; // Flush remaining messages

Sources: README.md:62-81, src/lib.rs:65-84

Architecture and Data Flow

SeaStreamer is architected as a set of sub-crates that separate core traits from specific backend implementations.

The diagram above illustrates how the sea-streamer-socket layer provides a backend-agnostic interface by abstracting specific implementations under the shared types defined in sea-streamer-types.

Sources: README.md:130-145, src/lib.rs:133-148

Backend Usage Examples

SeaStreamer allows switching backends at runtime using different URI schemes.

Kafka and Redis

Both Kafka and Redis backends are suitable for distributed production environments.

# Kafka
STREAMER_URI="kafka://localhost:9092"
# Redis
STREAMER_URI="redis://localhost:6379"

cargo run --bin producer -- --stream $STREAMER_URI/topic1

Sources: README.md:111-120, src/lib.rs:114-123

Stdio and File

Stdio is designed for unix pipes, while the File backend supports binary data and seeking via .ss (sea-stream) files.

The Stdio backend facilitates real-time processing through pipes, whereas the File backend uses a custom frame format with "Beacons" at fixed intervals (default 1MB) to allow efficient seeking.

Sources: README.md:122-128, sea-streamer-file/src/lib.rs:24-34, src/lib.rs:125-131

Conclusion

SeaStreamer provides a scalable and testable framework for Rust stream processing. By abstracting backends like Kafka and Redis behind a common interface and providing local-first tools like Stdio and File backends, it simplifies the transition from local development to production deployment.

Sources: README.md:16-25, src/lib.rs:18-27