Kafka & Redpanda Backend
Relevant source files
The following files were used as context for generating this wiki page:
The sea-streamer-kafka crate provides a robust, async-native implementation for Kafka and Redpanda within the SeaStreamer ecosystem. It acts as a specialized backend that wraps the synchronous rdkafka library into a type-safe, asynchronous API, ensuring that stream processing remains non-blocking and safe from common race conditions.
This backend is integrated into the broader project via the sea-streamer-socket layer, allowing users to switch between Kafka and other backends like Redis or Stdio at runtime without changing application logic. It supports essential streaming operations including time-based seeking, offset rewinding, and explicit message commits.
Sources: sea-streamer-kafka/src/lib.rs:1-25, README.md:143-157, sea-streamer-socket/src/lib.rs:1-10
Architecture and Core Components
The Kafka backend is built as a wrapper around rdkafka, which interfaces with the C-based librdkafka. SeaStreamer abstracts these details into high-level Rust components: KafkaStreamer, KafkaConsumer, and KafkaProducer.
Component Overview
| Component | Description |
|---|---|
KafkaStreamer | Manages the connection to the cluster and handles lifecycle events like flushing producers on disconnect. |
KafkaConsumer | Handles message ingestion, supporting group coordination and manual offset management. |
KafkaProducer | Manages message egress, providing receipts for sent messages and flushing capabilities. |
KafkaConsumerOptions | Provides typed parameters for configuring consumer behavior. |
Sources: sea-streamer-kafka/README.md:1-20, sea-streamer-kafka/src/lib.rs:1-15
Connection Hierarchy
The following diagram illustrates the relationship between the SeaStreamer abstraction and the underlying Kafka client:
A visual representation of the dependency stack for the Kafka backend.
Sources: sea-streamer-kafka/Cargo.toml:34-40, README.md:155-157
Consumer Logic
The KafkaConsumer is designed for flexibility in how data is retrieved and managed. It supports several advanced operations that are properly wrapped in async interfaces.
Consumer Operations
- Seeking: Consumers can
seekto a specific point in time orrewindto a particular offset. - Committing: Messages can be explicitly committed to mark progress within a consumer group.
- Safety: Methods are marked with
&mutto eliminate potential race conditions during state changes.
Sources: sea-streamer-kafka/README.md:10-15, sea-streamer-kafka/src/lib.rs:11-14
Consumer Initialization Flow
The sequence of creating and using a Kafka consumer within the SeaStreamer framework.
Sources: README.md:46-60, sea-streamer-kafka/README.md:10-14
Producer Logic
The KafkaProducer provides a non-blocking interface for sending messages. It allows for different levels of delivery guarantees through the use of Receipts.
Key Features
- Async Send: Sending a message returns a future that can be
await-ed for aReceipt. - Discardable Receipts: If delivery confirmation is not required, the receipt can be discarded.
- Flushing: The producer supports explicit flushing to ensure all buffered messages are sent to the broker.
- Extended API: Support for
send_recordandsend_messagewas added in version 0.3.2 to provide more granular control.
Sources: sea-streamer-kafka/README.md:16-18, CHANGELOG.md:45-46
Configuration and Implementation Details
The backend utilizes rdkafka features to support various protocols and runtimes.
Feature Flags in Cargo.toml
The crate defines several features to control the inclusion of optional rdkafka capabilities:
| Feature | Description |
|---|---|
runtime-tokio | Enables integration with the Tokio async runtime. |
runtime-async-std | Enables integration with the async-std runtime. |
ssl / ssl-vendored | Enables SSL support for secure cluster connections. |
sasl-vendored | Enables SASL authentication support. |
libz / libz-static | Enables zlib compression for message payloads. |
Sources: sea-streamer-kafka/Cargo.toml:43-55, CHANGELOG.md:103-104
Network Defaults
- Default Port: The backend assumes port
9092for Kafka/Redpanda connections if none is specified. - Default Timeout: A default timeout of 60 seconds is applied to operations where a timeout is required but not provided.
Sources: sea-streamer-kafka/src/lib.rs:32-35
Conclusion
The Kafka & Redpanda backend for SeaStreamer provides a high-level, safe, and asynchronous interface to Kafka clusters. By leveraging rdkafka and integrating with the sea-streamer-socket API, it allows developers to build scalable stream processors that benefit from Rust's type system and async performance while remaining backend-agnostic.
Sources: sea-streamer-kafka/README.md:1-25, README.md:143-157