Feature: Durable Storage

Relevant source files

The following files were used as context for generating this wiki page:

• apps/runtime/crates/core/src/store.rs
• README.md
• apps/runtime/crates/core/src/config.rs
• apps/runtime/crates/ts-addon/src/lib.rs
• apps/runtime/crates/core/examples/improved.rs
• apps/runtime/crates/core/src/workflow.rs
• apps/api/examples/nodejs-usage.js

The Durable Storage system is a core feature of Kodus Runtime, providing a persistent, high-performance storage layer for events. It is built on top of the Sled embedded database, which offers transactional, log-structured storage optimized for modern hardware. This system ensures that events are safely stored on disk and can be efficiently retrieved for processing, even in the case of application restarts or failures.

The storage layer supports asynchronous operations, batch processing for improved throughput, and advanced search capabilities. Events can be queried by their source or correlation ID, facilitating event tracking and debugging. The entire storage mechanism is configurable through the project's central config.json file.

Sources: README.md:10, apps/runtime/crates/core/src/store.rs

Architecture

The primary component of the durable storage system is the DurableStore struct. It encapsulates all interactions with the underlying Sled database, providing a high-level API for event persistence and retrieval.

Sources: apps/runtime/crates/core/src/store.rs:16-22

Core Components

The DurableStore manages three separate Sled "trees" (analogous to tables in a traditional database) to store and index event data:

1. events: The main storage for event data. It maps a unique Uuid to the serialized Event object.
2. source_idx: A secondary index that maps an event's source string to a list of Uuids. This allows for fast lookups of all events from a specific service or origin.
3. correlation_id_idx: Another secondary index that maps a correlation_id (Uuid) to a list of event Uuids. This is used to group and retrieve related events that are part of the same logical transaction or flow.

Sources: apps/runtime/crates/core/src/store.rs:18-20, apps/runtime/crates/core/src/store.rs:100-128

The following diagram illustrates the structure of the DurableStore and its relationship with the Sled database trees.

Configuration

The behavior of the DurableStore is configured via the store section in the main config.json file. These settings are defined in the StoreConfig struct.

Sources: apps/runtime/crates/core/src/config.rs:26-35, README.md:204-211

StoreConfig Options

Parameter	Type	Default Value	Description
data_dir	String	"data"	The directory where the Sled database files are stored.
batch_size	usize	1000	The number of events to process in a single batch operation.
compression_enabled	bool	false	Enables or disables data compression within Sled.
cache_enabled	bool	true	Enables an in-memory cache for the store.
max_cache_size	usize	10000	The maximum number of items to hold in the cache.
cleanup_old_snapshots	bool	true	If true, old database snapshots will be removed.
max_snapshots_to_keep	usize	10	The number of recent snapshots to retain.

Sources: apps/runtime/crates/core/src/config.rs:26-35, apps/runtime/crates/core/src/config.rs:81-93

Example config.json

{
  "store": {
    "data_dir": "data",
    "batch_size": 1000,
    "compression_enabled": false,
    "cache_enabled": true,
    "max_cache_size": 10000
  }
}

Example store configuration block in config.json.

Sources: README.md:204-211

Core Operations

Initialization

The store is initialized by calling DurableStore::open(path), which opens or creates the Sled database at the specified file path. Configuration methods like with_batch_size() can be chained to customize its behavior.

// From apps/runtime/crates/core/examples/improved.rs
let data_dir = config.get_data_dir()?.to_string_lossy().to_string();
let store = DurableStore::open(&data_dir)?
    .with_batch_size(config.store.batch_size)
    .with_compression(config.store.compression_enabled);

Sources: apps/runtime/crates/core/examples/improved.rs:38-42, apps/runtime/crates/core/src/store.rs:24-28

Event Persistence

Events are persisted using persist_event() for single events or persist_batch() for multiple events. The persistence process is transactional.

1. The Event is serialized into a byte vector.
2. The serialized event is inserted into the events tree with its Uuid as the key.
3. The source_idx tree is updated to associate the event's source with its Uuid.
4. If a correlation_id exists, the correlation_id_idx tree is also updated.

This flow is executed within a Sled transaction to ensure atomicity.

Sources: apps/runtime/crates/core/src/store.rs:59-98

The following sequence diagram illustrates the process for persisting a single event.

Event Retrieval

The DurableStore provides methods to search for events using the secondary indexes:

• find_by_source(source: &str): Retrieves all events associated with a given source. It looks up the list of event IDs in source_idx and then fetches each event from the main events tree.
• find_by_correlation_id(correlation_id: Uuid): Retrieves all events that share a correlation ID. It follows a similar process using the correlation_id_idx.

Sources: apps/runtime/crates/core/src/store.rs:100-128

This flowchart shows the logic for finding events by source.

Node.js Integration

The durable storage functionality is exposed to the Node.js environment through a native addon created with N-API. The KodusRuntime class in the addon wraps DurableStore and provides an asynchronous API for JavaScript/TypeScript applications.

Sources: apps/runtime/crates/ts-addon/src/lib.rs:7-14

Exposed Methods

The KodusRuntime class exposes the following methods related to storage:

Method	Description
new KodusRuntime(data_dir: string)	Constructor that initializes the DurableStore at the given path.
persistEvent(payload: Buffer, source: string)	Persists a simple event.
persistEventWithMetadata(...)	Persists an event with full metadata, including priority, user ID, correlation ID, and tags.
findEventsBySource(source: string)	Finds all events matching the given source.
getStats()	Retrieves statistics from the store, such as the total number of events.

Sources: apps/runtime/crates/ts-addon/src/lib.rs:16-114, apps/api/examples/nodejs-usage.js

Example Usage (JavaScript)

// From apps/api/examples/nodejs-usage.js
const { KodusRuntime } = require('../crates/ts-addon');

// 1. Initialize the runtime
const runtime = new KodusRuntime('./data');

// 2. Persist event with metadata
const eventId = await runtime.persistEventWithMetadata(
    Buffer.from('{"payment": {"amount": 100.50}}'),
    'payment-service',
    2, // Priority: High
    'user-123',
    'correlation-456',
    [['environment', 'production']]
);

// 3. Find events
const paymentEvents = await runtime.findEventsBySource('payment-service');

Sources: apps/api/examples/nodejs-usage.js:3-28

Other Integrations

The DurableStore is a fundamental building block used by other parts of the system, such as the WorkflowEngine, which uses it to manage and persist workflow state.

Summary

The Durable Storage feature provides a robust and efficient foundation for the Kodus Runtime event processing system. By leveraging the Sled embedded database, it guarantees that events are persisted transactionally and can be indexed for fast retrieval. Its clear separation of concerns, comprehensive configuration, and exposure to both Rust and Node.js environments make it a versatile and critical component of the overall architecture.