Feature: Batch Processing
Relevant source files
The following files were used as context for generating this wiki page:
Batch processing is a core performance feature of the Kodus Runtime, designed to improve the efficiency of event persistence. It allows multiple events to be written to the DurableStore in a single, atomic operation, reducing I/O overhead and increasing throughput. This capability is primarily implemented within the Rust kodus-core crate.
The system supports both explicit batching, where a developer can call a specific function to persist a collection of events, and configurable internal batching within the storage engine. This feature is crucial for high-load scenarios where a large volume of events needs to be ingested and stored reliably.
Sources: README.md:14, apps/runtime/crates/core/examples/improved.rs:67-69
Core Architecture
The batch processing logic is centralized in the DurableStore component of the Rust core. The primary method for this is persist_batch, which takes a vector of Event objects and writes them to the underlying Sled-based storage.
Sources: apps/runtime/crates/core/examples/improved.rs:68, README.md:210
Data Flow
The following diagram illustrates the flow of persisting multiple events using the batching feature.
This diagram shows a client application collecting events into a vector and passing them to the DurableStore for persistence.
Sources: apps/runtime/crates/core/examples/improved.rs:51-69
Key Components
| Component | Description | Relevant File |
|---|---|---|
DurableStore | The main storage struct that provides the persist_batch method for writing events. | apps/runtime/crates/core/examples/improved.rs:38 |
Event | The fundamental data structure representing a single event with its payload and metadata. Batches are collections of Events. | apps/runtime/crates/core/src/queue.rs |
persist_batch | The asynchronous function on DurableStore that accepts Vec<Event> and persists them. | apps/runtime/crates/core/examples/improved.rs:68 |
Configuration
The behavior of the storage layer, including aspects related to batching, can be configured via a config.json file. The store.batch_size parameter controls the size of internal batches for operations within the store.
Sources: README.md:219-224
Store Configuration Parameters
| Parameter | Type | Description | Default Value |
|---|---|---|---|
batch_size | integer | The number of events to group in an internal batch operation. | 1000 |
Sources: README.md:220
An example config.json snippet for the store:
{
"store": {
"data_dir": "data",
"batch_size": 1000,
"compression_enabled": false,
"cache_enabled": true,
"max_cache_size": 10000
}
}
Sources: README.md:218-225
The DurableStore can be initialized with this batch size.
let store = DurableStore::open(&data_dir)?
.with_batch_size(config.store.batch_size);
Sources: apps/runtime/crates/core/examples/improved.rs:38-40
Usage
Rust Implementation
In Rust, batch processing is achieved by creating a Vec<Event> and passing it to the store.persist_batch() method. This is the most direct way to leverage the batching feature for maximum performance.
The following sequence diagram shows the interaction.
Code Example
This example demonstrates creating two events and persisting them in a single batch call.
// apps/runtime/crates/core/examples/improved.rs:51-70
// 4. Criar eventos com metadata rica
let correlation_id = Uuid::new_v4();
let mut events = Vec::new();
// Evento de alta prioridade
let high_priority_event = Event::new(
b"critical data".to_vec(),
"payment-service".to_string(),
)
.with_priority(Priority::High)
.with_correlation_id(correlation_id)
.with_user_id("user-123".to_string())
.with_tag("environment".to_string(), "production".to_string())
.with_tag("region".to_string(), "us-east-1".to_string());
// Evento normal
let normal_event = Event::new(
b"regular data".to_vec(),
"user-service".to_string(),
)
.with_priority(Priority::Normal)
.with_correlation_id(correlation_id)
.with_user_id("user-456".to_string());
events.push(high_priority_event);
events.push(normal_event);
// 5. Processar eventos em batch
println!("📦 Processando {} eventos em batch...", events.len());
let ids = store.persist_batch(events).await?;
println!("✅ {} eventos persistidos com IDs: {:?}", ids.len(), ids);
Sources: apps/runtime/crates/core/examples/improved.rs:51-70
Node.js / TypeScript Interface
The Node.js addon (kodus-runtime) provides a bridge to the core Rust functionality. However, based on the exposed API in ts-addon/src/lib.rs, there is no explicit batch persistence method available in the KodusRuntime class for Node.js users.
The available methods are for single-event persistence:
persist_event(payload, source)persist_event_with_metadata(...)
Sources: apps/runtime/crates/ts-addon/src/lib.rs:29-65
Usage examples for Node.js confirm this, showing events being persisted one by one in a loop rather than as a single batch.
// apps/api/examples/nodejs-usage.ts:60-74
const paymentEventIds: string[] = [];
for (const payment of paymentEvents) {
const eventData = createPaymentEvent(payment);
const eventId = await runtime.persistEventWithMetadata(
eventData.payload,
eventData.source,
eventData.priority,
eventData.userId,
eventData.correlationId,
eventData.tags
);
paymentEventIds.push(eventId);
console.log(`✅ Pagamento de ${payment.amount} ${payment.currency} persistido: ${eventId}`);
}
This code iterates through payments and calls persistEventWithMetadata for each one, indicating the absence of a batch alternative at the TypeScript level.
Sources: apps/api/examples/nodejs-usage.ts:60-74
Summary
Batch processing is a powerful, performance-oriented feature within the kodus-runtime's Rust core. It is exposed via the DurableStore::persist_batch method and can be tuned with the store.batch_size configuration. While it is a key feature for backend performance, it is not currently exposed through the Node.js/TypeScript wrapper, which only supports single-event persistence.
Sources: README.md:14, apps/runtime/crates/core/examples/improved.rs:68, apps/runtime/crates/ts-addon/src/lib.rs:29-65