Consumers: Modes, Seeking & Reset

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• sea-streamer-types/src/lib.rs
• sea-streamer-redis/tests/seek-rewind.rs
• sea-streamer-file/src/lib.rs
• sea-streamer-file/tests/loopback.rs
• sea-streamer-file/tests/source.rs

SeaStreamer provides a unified abstraction for consuming messages from various backends like Redis, Kafka, Stdio, and Files. The consumer interface is designed to handle different streaming behaviors through the use of ConsumerMode, SeaStreamReset (or AutoStreamReset), and explicit seeking/rewinding capabilities. This allows developers to build backend-agnostic stream processors that can either process real-time data, resume from the last committed point, or load-balance messages across multiple workers.

Sources: README.md:129-136, sea-streamer-redis/tests/seek-rewind.rs:18-24

Consumer Modes

SeaStreamer defines several primary consumption modes that dictate how a consumer interacts with the stream and other members of a consumer group.

RealTime Mode

In RealTime mode, the consumer is primarily interested in new messages arriving in the stream. If the stream is new to the consumer, it uses the AutoStreamReset setting to determine the starting point (e.g., the earliest available message or the latest).

Resumable Mode

Resumable mode is used when a consumer needs to pick up where it left off. This mode typically involves a ConsumerGroup and utilizes server-side or local state to track the last acknowledged message. In Redis, this involves managing consumer group offsets, while in the File backend, it is listed as a future enhancement that might use SQLite for local state.

LoadBalanced Mode

This mode allows multiple consumers within the same group to share the processing load. Messages are distributed among group members. In Stdio and File backends, this is implemented using a round-robin distribution strategy.

Sources: README.md:195-205, README.md:270-275, sea-streamer-redis/tests/seek-rewind.rs:27-33

Mode	Backend Support (Example)	Behavior
RealTime	Redis, Kafka, File, Stdio	Tracks latest messages; uses Reset policy if new.
Resumable	Redis, Kafka	Resumes from last committed/ACKed offset.
LoadBalanced	Stdio, File, Redis	Distributes messages across a group (e.g., round-robin).

Sources: README.md:198-210, sea-streamer-file/src/lib.rs:25-27

Stream Reset Policies

When a consumer starts and no previous offset is found (or the current offset is invalid), the AutoStreamReset policy determines the behavior.

• Earliest: The consumer starts reading from the very beginning of the stream.
• Latest: The consumer starts reading only new messages that arrive after the consumer has connected.

Sources: README.md:55-56, sea-streamer-redis/tests/seek-rewind.rs:77-78

Reset Flow Logic

The following diagram illustrates how the consumer determines its starting position based on mode and reset policy:

The diagram shows the decision path for initial stream positioning.
Sources: README.md:53-60, sea-streamer-redis/tests/seek-rewind.rs:73-85

Seeking and Rewinding

SeaStreamer provides a powerful API for moving the consumer's "cursor" within a stream. This is essential for replaying data or jumping to specific events.

Seek Targets

Consumers can seek based on different criteria:

1. Sequence Number (SeqNo): Jumping to a specific, unique identifier for a message.
2. Timestamp: Jumping to the first message at or after a specific point in time.
3. Logical Positions: Using SeqPos::Beginning to jump to the start or SeqPos::End to jump to the end (tail) of the stream.

Backend Implementation Details

• Redis: Supports seeking by time and rewinding to specific offsets. Redis sequence numbers are not contiguous.
• Kafka: Supports seeking by time and rewinding to contiguous offsets.
• File: Uses "Beacons" (inplace indices) at fixed intervals (default 1MB) to allow efficient fast-forwarding and seeking within binary .ss files without reading every message.

Sources: README.md:179-183, README.md:203-205, sea-streamer-file/src/lib.rs:18-24, sea-streamer-redis/tests/seek-rewind.rs:94-106

Seek and Rewind Sequence

The sequence diagram illustrates the interaction between the application and the backend during seek operations.
Sources: sea-streamer-redis/tests/seek-rewind.rs:87-106, sea-streamer-file/tests/source.rs:96-105

Implementation Example: Redis Seek/Rewind

In a typical test scenario, a consumer can be moved back and forth to verify data consistency.

// Create consumer with Earliest reset
let mut options = RedisConsumerOptions::new(ConsumerMode::RealTime);
options.set_auto_stream_reset(AutoStreamReset::Earliest);
let mut seeker = streamer.create_consumer(&[stream], options).await?;

// Rewind to beginning
seeker.rewind(SeqPos::Beginning).await?;

// Seek to a specific timestamp captured earlier
seeker.seek(point_in_time).await?;

// Fast-forward to the end to catch new messages
seeker.rewind(SeqPos::End).await?;

Sources: sea-streamer-redis/tests/seek-rewind.rs:76-110

The Role of Beacons in File Seeking

For the File backend, the sea-streamer-file format is optimized for seeking. Instead of delimiters, it uses fixed-interval Beacons. A Beacon acts as an inplace index that summarizes the streams at that point in the file, allowing the MessageSource to align with message boundaries and jump to specific timestamps or sequence numbers efficiently.

Sources: sea-streamer-file/src/lib.rs:18-24, sea-streamer-file/tests/loopback.rs:103-118

Summary

The consumer system in SeaStreamer balances backend-specific capabilities with a high-level, generic interface. By utilizing ConsumerMode and AutoStreamReset, developers can define the durability and distribution of their streams. The addition of explicit seek and rewind methods ensures that applications can handle complex replay scenarios, whether interacting with a distributed cluster like Kafka/Redis or a local binary stream file.

Sources: README.md:129-145, sea-streamer-file/src/lib.rs:25-30