Utility: Redis Streams Dump

Relevant source files

The following files were used as context for generating this wiki page:

• sea-streamer-redis/redis-streams-dump/Cargo.toml
• sea-streamer-redis/redis-streams-dump/src/main.rs
• sea-streamer-redis/README.md
• sea-streamer-redis/src/lib.rs
• sea-streamer-file/src/lib.rs
• sea-streamer-types/Cargo.toml

The Redis Streams Dump utility is a specialized tool designed to bridge the gap between volatile Redis Streams and persistent storage. Its primary purpose is to export (or "dump") the content of a Redis Stream into a SeaStreamer binary file (.ss), enabling data archival, offline analysis, or migration between different SeaStreamer backends.

Within the SeaStreamer ecosystem, this utility demonstrates the power of the generic trait system by consuming messages via the sea-streamer-redis backend and sinking them into the sea-streamer-file backend. It supports time-range filtering, allowing users to extract specific segments of a stream based on timestamps.

Sources: sea-streamer-redis/redis-streams-dump/Cargo.toml:8, sea-streamer-redis/README.md:92, sea-streamer-redis/redis-streams-dump/src/main.rs:16-30

System Architecture and Data Flow

The utility operates as a single-threaded asynchronous process (leveraging the Tokio runtime) that manages a unidirectional data flow from a Redis server to a local file.

Data Migration Flow

1. Connection: Establishes a connection to the Redis server using a StreamUrl.
2. Consumer Creation: Initializes a RedisConsumer in RealTime mode but configured to start from the earliest available message.
3. Range Seeking: If a start time is provided, the consumer performs a seek operation to align the stream pointer.
4. Message Processing: Iteratively fetches messages, converts them to OwnedMessage types, and writes them to a MessageSink.
5. Finalization: Flushes the file sink to ensure data integrity before exiting.

The following diagram illustrates the internal processing logic:

The diagram shows the sequential steps from argument parsing to the message extraction loop and final flushing.

Sources: sea-streamer-redis/redis-streams-dump/src/main.rs:33-72, sea-streamer-redis/README.md:92

Component Specifications

Command Line Interface (CLI)

The utility uses clap to provide a robust CLI for defining the source stream and destination file.

Argument	Type	Requirement	Description
--stream	StreamUrl	Required	The Redis URI with stream key (e.g., redis://localhost/my_stream)
--output	FileId	Required	Path to the output SeaStreamer file (.ss). Overwrites existing files.
--since	String	Optional	Start of range (format defined by TIMESTAMP_FORMAT)
--until	String	Optional	End of range (format defined by TIMESTAMP_FORMAT)

Sources: sea-streamer-redis/redis-streams-dump/src/main.rs:16-30

Core Components and Logic

The implementation relies on several key structures from the SeaStreamer library:

• RedisStreamer: Used to establish the initial connection to the Redis instance.
• RedisConsumerOptions: Configured with ConsumerMode::RealTime and AutoStreamReset::Earliest to ensure the dump begins at the start of the stream if no offset is specified.
• MessageSink: A sea-streamer-file component that handles the binary writing, beaconing, and file size limits.
• PrimitiveDateTime: Used for parsing the --since and --until strings into UTC timestamps.

Default Configurations

The utility employs standard SeaStreamer file defaults for the output:

• Beacon Interval: 1MB (defines how often index markers are placed in the file).
• File Size Limit: 16GB.
• Read Timeout: 1 second (the utility stops if no message is received within this duration).

Sources: sea-streamer-redis/redis-streams-dump/src/main.rs:43-58, sea-streamer-file/src/lib.rs:98-99, sea-streamer-redis/src/lib.rs:25

Sequence Diagram: Extraction Process

This sequence diagram details the interactions between the utility, the Redis backend, and the File backend during a dump operation.

The diagram depicts the synchronous-like flow of the asynchronous loop where messages are fetched from Redis and immediately written to the local file system.

Sources: sea-streamer-redis/redis-streams-dump/src/main.rs:60-75, sea-streamer-redis/README.md:92

Implementation Snippet: The Extraction Loop

The heart of the utility is the while let loop which handles the message transfer and the --until time constraint.

// From sea-streamer-redis/redis-streams-dump/src/main.rs
let mut count = 0;
while let Ok(Ok(mess)) = tokio::time::timeout(dur, consumer.next()).await {
    if let Some(until) = &until {
        if &mess.timestamp() > until {
            break;
        }
    }
    sink.write(mess.to_owned_message())?;
    count += 1;
}

Sources: sea-streamer-redis/redis-streams-dump/src/main.rs:64-73

Summary

The Redis Streams Dump utility provides a critical maintenance and migration path for users of the Redis backend. By leveraging the sea-streamer-file backend's MessageSink, it creates high-performance binary archives that remain compatible with the rest of the SeaStreamer ecosystem, including the ss-decode tool and the File backend consumers.