Core Traits & Types

Relevant source files

The following files were used as context for generating this wiki page:

• sea-streamer-types/src/lib.rs
• sea-streamer-socket/src/lib.rs
• src/lib.rs
• README.md
• sea-streamer-types/Cargo.toml

The SeaStreamer ecosystem is built upon a foundation of generic traits and common types defined in the sea-streamer-types crate. This design allows the toolkit to remain backend-agnostic, enabling developers to write stream processing logic that can interact with various streaming servers like Kafka, Redis, or local files without changing the application code.

The primary purpose of these core types is to provide an async, non-blocking API that facilitates real-time stream processing. By abstracting the specific implementation details of each backend into a common interface, SeaStreamer ensures high concurrency and testability across different environments.

Sources: sea-streamer-types/src/lib.rs:1-5, README.md:12-25

System Architecture

SeaStreamer's architecture is modular, with sea-streamer-types serving as the base crate that defines the interfaces. Higher-level crates like sea-streamer-socket provide a concrete-type API (e.g., SeaStreamer, SeaProducer, SeaConsumer) that allows backend selection at runtime.

Hierarchy of Components

The following diagram illustrates the relationship between the core traits and the backend-specific implementations.

The diagram shows how the User Application interacts with the Backend-agnostic Socket API, which is built upon the Core Traits, eventually dispatching to specific backend implementations.

Sources: README.md:120-135, sea-streamer-socket/src/lib.rs:5-15

Core Traits & Abstractions

The system revolves around four primary abstractions: the Streamer, the Producer, the Consumer, and the Message. These are exported via sea-streamer-types and re-exported by the main sea-streamer crate.

Key Trait Definitions

Trait/Module	Description
Streamer	The entry point for connecting to a streaming server and creating producers or consumers.
Producer	Handles the transmission of messages to a specific stream.
Consumer	Handles the reception of messages from one or more streams.
Message	Represents the data unit, including payload, timestamp, and sequence information.
Options	Configuration structures for connections, consumers, and producers.

Sources: sea-streamer-types/src/lib.rs:13-21, src/lib.rs:180-185

Connection and Lifecycle

The interaction typically begins by establishing a connection through a Streamer. This connection is used to spawn specialized actors for sending or receiving data.

This sequence shows the typical lifecycle from connection establishment to message production and consumption.

Sources: README.md:46-75, sea-streamer-socket/src/lib.rs:1-10

Data Structures and Types

The toolkit utilizes a strict type system to ensure safety and clarity when handling stream metadata.

Message Structure

A Message in SeaStreamer is more than just a payload; it contains vital metadata required for stream processing:

• Timestamp: The time the message was produced or received.
• Stream Key: The identifier for the stream (e.g., a Kafka topic or Redis key).
• Sequence Number: A unique identifier for the message's position in the stream.
• Shard ID: Identifies the sub-partition of the stream.
• Payload: The actual data, which can be binary or UTF-8 strings.

Sources: README.md:215-240, sea-streamer-types/Cargo.toml:22-26

Stream Identifiers

SeaStreamer uses specific URI formats to identify backends and streams at runtime:

• StreamerUri: Represents the server address (e.g., kafka://localhost:9092).
• StreamKey: A string identifier for a specific channel of data.

// Examples of URI formats supported by the core types
// redis://localhost:6379/stream_name
// kafka://localhost:9092/topic_name
// file:///path/to/file.ss/stream_name
// stdio:///stream_name

Sources: README.md:95-115, sea-streamer-socket/src/lib.rs:32-50

Runtime & Feature Support

The core types are designed to work across different async runtimes and support optional features like JSON serialization.

• Async Runtimes: Supports both tokio and async-std through the sea-streamer-runtime abstraction.
• Serialization: If the json feature is enabled, types integrate with serde and serde_json for easy payload manipulation.
• Wide Sequence Numbers: The wide-seq-no feature allows for larger sequence identifiers.

Sources: sea-streamer-types/Cargo.toml:19-27, src/lib.rs:200-210

Conclusion

The Core Traits & Types form the skeletal structure of SeaStreamer. By defining a rigorous yet flexible interface for producers, consumers, and streamers, the project enables a "write once, stream anywhere" development experience. This architecture is what allows the sea-streamer-socket crate to provide a truly backend-agnostic API that can be configured entirely through URIs and runtime options.

Sources: README.md:140-150, sea-streamer-socket/src/lib.rs:1-10