Local Testing & Loopbacks

Relevant source files

The following files were used as context for generating this wiki page:

• sea-streamer-file/tests/loopback.rs
• README.md
• src/lib.rs
• sea-streamer-file/src/lib.rs
• sea-streamer-stdio/Cargo.toml

SeaStreamer provides specialized backends for local development and testing, specifically through the stdio and file modules. These tools allow developers to simulate stream processing environments without the overhead of deploying full Kafka or Redis clusters. By utilizing Unix pipes and local files, SeaStreamer enables high-speed "loopback" testing where data is produced and consumed within the same local environment or even the same process.

These local backends implement the same generic trait interfaces as the production-grade backends, ensuring that code tested locally can be transitioned to cloud environments with minimal changes. The system supports both real-time data flow (via Stdio) and replayable streams (via File), which are critical for debugging complex stream logic.

Sources: README.md:14-23, src/lib.rs:24-33

Local Testing Architecture

The local testing architecture relies on backend-agnostic interfaces provided by sea-streamer-socket and implemented specifically for local IO in the stdio and file crates.

Stdio Loopbacks

The sea-streamer-stdio backend treats the standard input and output as a stream. It is designed for real-time interaction where processors can be chained using standard Unix pipes.

File-Based Loopbacks

The sea-streamer-file backend extends local testing by providing persistence. It supports "LiveReplay" mode, allowing a consumer to read from the beginning of a file while a producer is actively appending to it. This creates a durable loopback mechanism.

Key Components and Data Structures

Testing involves several low-level structures that define the file format and communication protocols for local backends.

Loopback Components

Component	Description
FileSink	Writes binary data to a local .ss file, handling flushes and file limits.
FileSource	Reads data from a file, supporting seeking to timestamps or sequence numbers.
AsyncFile	An asynchronous file handle abstraction used by sinks and sources.
MessageSink	High-level component for writing OwnedMessage objects to files.
MessageSource	High-level component for reading messages with support for different StreamMode settings.

Sources: sea-streamer-file/tests/loopback.rs:18-24, sea-streamer-file/src/lib.rs:59-75

Binary Format Structures

To support efficient local replay and seeking, the file backend utilizes a specific binary structure:

• HeaderV1: Contains metadata including file name, creation timestamp, and beacon intervals.
• Beacon: Periodic index entries that summarize stream state to allow fast-forwarding and seeking.
• Message: The wrapper for the actual payload, including a checksum for data integrity.

Sources: sea-streamer-file/tests/loopback.rs:58-95, sea-streamer-file/src/lib.rs:18-23

Loopback Testing Logic

The primary method for verifying stream logic locally is the loopback test, which ensures that data written by a producer is accurately retrieved by a consumer.

Seeking and Rewinding

Unlike standard pipes, file-based loopbacks allow for non-linear access. This is tested using SeqPos, which allows a source to jump to specific points in the stream.

Implementation Example: Basic Loopback

In a test environment, a loopback is established by creating a sink and source on the same temporary file path.

let path = temp_file("loopback-test")?;
let mut sink = FileSink::new(AsyncFile::new_ow(path.clone()).await?, LIMIT)?;
let mut source = FileSource::new(path.clone(), ReadFrom::Beginning).await?;

let bytes = Bytes::from_bytes(vec![1, 2, 3, 4]);
bytes.write_to(&mut sink)?;
sink.flush(1).await?;

let read = Bytes::read_from(&mut source, bytes.len()).await?;
assert_eq!(read, bytes);

Sources: sea-streamer-file/tests/loopback.rs:25-38

Summary of Local Testing Backends

Feature	Stdio Backend	File Backend
Data Type	UTF-8 Text	Binary / UTF-8
Persistence	Volatile (Pipes)	Persistent (.ss files)
Access Pattern	Sequential Only	Sequential & Seeking
Concurrency	One Streamer per process	Multiple Streamers per process
Primary Use	CLI piping / Real-time	Debugging / Replay / Benchmarking

Sources: README.md:243-255, sea-streamer-file/src/lib.rs:5-15

Local testing and loopbacks in SeaStreamer provide a robust framework for developing stream processors. By leveraging the stdio and file backends, developers can validate complex behaviors—such as message ordering, seeking, and error handling—entirely within a local environment, ensuring high confidence before deploying to distributed systems.