Async Runtime Abstraction
Relevant source files
The following files were used as context for generating this wiki page:
The sea-streamer-runtime crate serves as a thin abstraction layer designed to unify the API surfaces of popular Rust asynchronous runtimes, specifically tokio and async-std. By aligning type signatures and providing a common set of functions, it enables the broader SeaStreamer ecosystem to remain runtime-agnostic, allowing developers to build stream processing applications that can switch between runtimes without significant code changes.
This module is a core component of the SeaStreamer architecture, ensuring that high-level backend implementations—such as Redis, Kafka, and File—can operate efficiently regardless of the underlying executor.
Sources: sea-streamer-runtime/README.md:1-5, README.md:173-176
Runtime Configuration and Dependency Management
The abstraction is primarily controlled through Cargo features. This allows the compiler to include only the necessary dependencies for the target runtime, keeping the binary footprint minimal.
Supported Runtimes
The library supports the following primary features for runtime selection:
| Feature | Dependency | Description |
|---|---|---|
runtime-tokio | tokio (v1.10+) | Enables support for the Tokio runtime, including time and sync primitives. |
runtime-async-std | async-std (v1.12+) | Enables support for the async-std runtime. |
file | tokio/fs, tokio/io-util | Optional extension for file-system specific async operations. |
Sources: sea-streamer-runtime/Cargo.toml:22-29, Cargo.toml:44-47
Implementation Selection Logic
The workspace-level Cargo.toml propagates these runtime choices down to sub-crates like sea-streamer-socket and sea-streamer-kafka. This ensures that if a user selects runtime-tokio at the top level, the entire stack uses the Tokio-based implementation of the abstraction.
Architecture and Components
The abstraction provides standardized interfaces for common asynchronous tasks, including task spawning and synchronization primitives.
Task Management
While the specific internal task module files were not fully expanded in the context, the library provides a unified interface for spawning and managing asynchronous tasks. This allows the backends to perform background processing—such as Redis XREAD loops or Kafka heartbeat management—in a way that is compatible with the host application's executor.
Sources: README.md:129-132, sea-streamer-runtime/Cargo.toml:23-24
Synchronization Primitives
The abstraction includes support for mutexes and other synchronization tools that work across different runtimes. In SeaStreamer, these are used sparingly on the "hot path" to maintain high performance, with concurrency often achieved through message passing (e.g., using flume channels).
Sources: README.md:11-13, README.md:129-132
Integration with Backends
The sea-streamer-runtime crate is a dependency for all major SeaStreamer backends. This allows backend-specific features to be conditionally compiled based on the runtime.
Backend Runtime Support
| Backend | Runtime Abstraction Usage |
|---|---|
| Kafka | Wraps synchronous rdkafka calls into async tasks compatible with the selected runtime. |
| Redis | Manages background read/write loops to achieve high throughput (100k+ msg/s). |
| File | Utilizes runtime-specific filesystem IO (e.g., tokio::fs). |
| Stdio | Handles real-time streams via standard input/output pipes. |
Sources: README.md:100-105, README.md:129-136, sea-streamer-runtime/Cargo.toml:29
Summary
The Async Runtime Abstraction provides a critical layer of flexibility for the SeaStreamer toolkit. By standardizing the interface for task execution and synchronization, it allows the project to support diverse environments—from high-performance microservices using Tokio to applications preferring the async-std ecosystem—without duplicating backend logic. This design supports the project's goal of being a generic, testable, and micro-service oriented toolkit for real-time stream processing.
Sources: README.md:10-25, sea-streamer-runtime/README.md:1-5