Stream Sharding & Load Balancing

Relevant source files

The following files were used as context for generating this wiki page:

• sea-streamer-redis/src/lib.rs
• sea-streamer-stdio/src/lib.rs (derived from workspace structure)
• sea-streamer-file/src/lib.rs
• README.md
• sea-streamer-types/src/lib.rs (derived from workspace structure)

Stream sharding and load balancing are core features in SeaStreamer that allow for horizontal scaling and efficient message distribution across multiple consumers. These mechanisms ensure that high-throughput streams can be processed in parallel while maintaining specific delivery guarantees based on the chosen backend and consumer mode.

In SeaStreamer, sharding is handled differently across backends. For instance, Redis provides basic stream sharding by splitting a stream into multiple sub-streams, while Kafka maintains a strict 1-to-1 mapping between consumers and shards within a consumer group. Backends like Stdio and File implement load balancing via a round-robin distribution among consumers in the same group.

Sources: README.md:144-150, sea-streamer-redis/src/lib.rs:30-35, sea-streamer-file/src/lib.rs:24-26

Backend Sharding Implementations

SeaStreamer abstracts the sharding logic to provide a consistent interface while respecting the underlying infrastructure of the selected backend.

Redis Sharding & Distribution

In the Redis backend, sharding allows for splitting a single stream into multiple sub-streams. Unlike Kafka, Redis sequence numbers are not contiguous, and messages are dispatched to group members in a "first-ask-first-served" manner. SeaStreamer builds a high-level API over Redis Streams to manage these complexities, offering a LoadBalanced mode with failover behavior.

Kafka Sharding

Kafka utilizes a contiguous sequence number system where shards (partitions) are strictly mapped to consumers within a consumer group. This ensures that only one consumer in a group processes a specific shard at any given time, providing ordered processing within that shard.

Sources: README.md:144-148

Load Balancing Mechanics

Load balancing in SeaStreamer is primarily achieved through ConsumerGroups. When multiple consumers share the same group ID, the system distributes incoming messages among them to prevent any single consumer from becoming a bottleneck.

Round-Robin Distribution (Stdio & File)

For the stdio and file backends, load balancing is implemented using a round-robin algorithm. This allows developers to spawn multiple async tasks to process messages in parallel locally.

Load Balancing Features by Backend

Backend	Sharding Method	Load Balancing Strategy	Sequence Contiguity
Redis	Sub-streams	First-ask-first-served / Failover	Non-contiguous
Kafka	Partitions	1-to-1 Shard Mapping	Contiguous
Stdio	N/A	Round-robin	N/A
File	Future Support*	Round-robin	N/A

*Note: Sharding in the File backend currently defaults to Shard ZERO.

Sources: README.md:144-155, sea-streamer-file/src/lib.rs:24-26, sea-streamer-file/src/lib.rs:80-82

Component Architecture

The sharding and load balancing logic is supported by the sea-streamer-types crate, which defines the common identifiers used for routing.

Key Data Structures

• ShardId: A unique identifier for a stream partition. In Redis, ZERO is often used as the default shard identifier.
• ConsumerGroup: A logical grouping of consumers used to trigger load-balancing logic.
• StreamKey: The identifier for the stream being sharded.

// Source: sea-streamer-redis/src/lib.rs:88
pub const ZERO: ShardId = ShardId::new(0);

// Source: sea-streamer-file/src/lib.rs:85
pub const SEA_STREAMER_WILDCARD: &str = "SEA_STREAMER_WILDCARD";

Sources: sea-streamer-redis/src/lib.rs:86-88, sea-streamer-file/src/lib.rs:84-85, README.md:218-220

Implementation Constraints

Currently, sharding implementation varies in maturity across backends:

1. Redis: Supports basic sharding but notes that sharding without Redis Cluster is limited. Shards and nodes have an M-N mapping in clusters.
2. File: Currently only streams to Shard ZERO; full sharding is listed in the project's TODO.
3. Stdio: Distributes messages assigned to broadcast or specific stream keys across consumers in a group.

Sources: sea-streamer-redis/src/lib.rs:55-60, sea-streamer-file/src/lib.rs:80-82, README.md:213-221

Conclusion

Stream Sharding and Load Balancing in SeaStreamer provide the necessary infrastructure for scalable, parallel processing of data streams. By utilizing ConsumerGroups for load balancing and backend-specific sharding (such as Kafka partitions or Redis sub-streams), SeaStreamer allows developers to build high-performance stream processors that can handle varying loads and ensure system reliability.