Implementing Custom Backends

Relevant source files

The following files were used as context for generating this wiki page:

• sea-streamer-types/src/lib.rs
• sea-streamer-socket/src/lib.rs
• sea-streamer-file/src/lib.rs
• sea-streamer-stdio/src/lib.rs
• README.md
• src/lib.rs

Implementing a custom backend in SeaStreamer involves adhering to a common set of traits and architectural patterns defined in sea-streamer-types. This abstraction allows developers to create backend-agnostic stream processors that can switch between different streaming technologies—such as Kafka, Redis, or local files—at runtime without code changes.

The toolkit is designed to be highly concurrent and non-blocking, supporting both tokio and async-std runtimes. Custom backends must implement core interfaces for streamers, producers, and consumers to integrate with the sea-streamer-socket layer, which provides the concrete-type API used by applications.

Sources: README.md:12-25, sea-streamer-socket/src/lib.rs:3-15

Architecture and Core Traits

SeaStreamer's backend architecture is built upon a hierarchy of crates. At the base, sea-streamer-types defines the protocol through Rust traits. Individual backend crates (e.g., sea-streamer-kafka, sea-streamer-redis) provide the implementation of these traits.

Primary Components

To implement a new backend, you must provide concrete implementations for the following abstractions:

• Streamer: The factory for creating producers and consumers. It manages the connection to the underlying streaming service.
• Producer: Handles sending messages to specific stream keys. It is typically non-blocking and may support flushing.
• Consumer: Handles receiving messages. It supports various modes like RealTime or LoadBalanced and may provide seeking/rewinding capabilities.

The diagram shows the relationship between the application, the socket abstraction layer, and the underlying backend implementations.

Sources: README.md:143-157, sea-streamer-types/src/lib.rs

Backend Implementation Comparison

Existing backends provide a template for how specific streaming technologies map to SeaStreamer traits.

Backend	Core Technology	Message Delivery	Sharding Support
Kafka	librdkafka	Contiguous sequence numbers	Shard-to-consumer 1:1
Redis	Redis Streams	First-ask-first-served	M-N Shard-to-node mapping
Stdio	Unix Pipes	Real-time UTF-8 lines	Round-robin load balancing
File	.ss binary format	Real-time & Replay	Fixed interval Beacons for seeking

Sources: README.md:175-185, README.md:200-215, sea-streamer-file/src/lib.rs:3-15

Implementing the Producer

Producers in SeaStreamer are designed to be "mutex-free" on the hot path. Implementation usually involves a background loop that handles the actual network I/O, while the send method pushes messages into a channel.

Producer Logic Flow

This diagram illustrates the non-blocking nature of the producer implementation where the application thread is decoupled from the network I/O.

Sources: README.md:225-235, sea-streamer-redis/README.md:45-55

Implementing the Consumer

Consumers must handle stream subscriptions and message retrieval. Depending on the backend, this might involve complex state management for offsets or acknowledgments.

Key Consumer Requirements:

1. Stream Selection: Ability to subscribe to multiple StreamKeys.
2. Message Framing: Translating raw backend data into SeaMessage objects containing timestamp, sequence, and payload.
3. Consumer Groups: Support for load balancing among multiple instances if applicable to the backend technology.

Example: Stdio Backend Framing

The stdio backend provides a simple text-based framing that custom backends can reference for lightweight implementations:
[timestamp | stream_key | sequence | shard_id] payload

Sources: README.md:265-275, sea-streamer-stdio/src/lib.rs:40-50

Integration with sea-streamer-socket

To make a custom backend accessible via the SeaStreamer entry point, it must be integrated into sea-streamer-socket. This crate provides the Backend enum and the logic to parse StreamerUri strings to select the correct implementation at runtime.

URI Parsing

Backends are identified by the scheme in the URI:

• kafka://... -> Kafka Backend
• redis://... -> Redis Backend
• stdio://... -> Stdio Backend
• file://... -> File Backend

Sources: sea-streamer-socket/src/lib.rs:40-60, README.md:159-170

Summary

Implementing a custom backend requires fulfilling the traits defined in sea-streamer-types while ensuring the implementation remains async and non-blocking. By following the pattern of separating I/O loops from processing loops (as seen in the Redis backend), developers can achieve high throughput and low latency. The final step is exposing the backend through the sea-streamer-socket URI-based connection mechanism to allow backend-agnostic application usage.

Sources: README.md:230-245, sea-streamer-socket/src/lib.rs:3-10