Storage and Buffering

Relevant source files

The following files were used as context for generating this wiki page:

• source/src/flb_storage.c
• source/src/flb_input_chunk.c
• source/lib/chunkio/src/chunkio.c
• source/lib/rocksdb/README.md
• source/src/flb_dump.c

The Storage and Buffering subsystem is a core component of FluentDo Agent, designed to provide a reliable and efficient mechanism for handling data between input and output plugins. It ensures data integrity and resilience by temporarily storing incoming records in chunks, which can be held in memory or persisted to the filesystem. This layer protects against data loss during high-load scenarios or temporary downstream unavailability.

The system is built upon the Chunk I/O (ChunkIO) library, which provides the underlying mechanics for chunk management. Each input plugin instance is associated with its own storage context and manages its own set of chunks. When a chunk is ready to be processed, it is encapsulated within a task and handed off to the engine for routing and delivery. The storage layer is extensible and can support different backends, such as the default filesystem or more advanced key-value stores like RocksDB.

Architecture Overview

The storage layer is composed of several key components that work together to manage the data pipeline. At the highest level, each input plugin instance has its own storage context. This context manages data chunks, which can be backed by memory or the filesystem.

The general data flow is as follows:

1. An input plugin receives data.
2. The data is appended to an active flb_input_chunk.
3. When a chunk is full or a flush is triggered, the chunk is marked as busy and is no longer available for new data.
4. The engine creates a flb_task that references the busy chunk.
5. The task is routed to the appropriate output plugins for delivery.
6. Once the data is successfully delivered, the task and its associated chunk are destroyed.

This flow is illustrated in the diagram below.

Core Components

Chunk I/O (ChunkIO) Library

Sources: source/lib/chunkio/src/chunkio.c

ChunkIO is the low-level library responsible for the creation, management, and I/O operations of data chunks. It provides an abstraction layer that allows chunks to be stored either in memory or on the filesystem, transparent to the higher-level components.

Chunk States

A chunk managed by ChunkIO can be in one of two primary states:

• Up: The chunk's content is loaded into memory. All memory-only chunks are permanently in the 'up' state. Filesystem-backed chunks are brought 'up' when they need to be processed.
• Down: The chunk's content is stored on the filesystem and is not currently loaded in memory. This helps to reduce memory consumption.

The following diagram shows the state transitions for a filesystem-backed chunk.

Key ChunkIO Functions

Function	Description
cio_chunk_create()	Creates a new chunk in the specified storage context.
cio_chunk_append()	Appends data to the chunk.
cio_chunk_is_up()	Checks if a chunk is currently loaded in memory ('up').
cio_chunk_get_content_size()	Returns the size of the data stored in the chunk.
cio_stats_get()	Retrieves statistics about the ChunkIO instance, such as the number of memory and filesystem chunks.

Sources: source/lib/chunkio/src/chunkio.c

Input Chunks (flb_input_chunk)

Sources: source/src/flb_input_chunk.c

The flb_input_chunk is a structure that wraps a cio_chunk and links it to a specific input plugin instance. It serves as the primary buffer for incoming data.

Key fields of the flb_input_chunk structure:

Field	Description
chunk	A pointer to the underlying cio_chunk managed by the ChunkIO library.
busy	A boolean flag (FLB_TRUE/FLB_FALSE) indicating if the chunk is locked for flushing. No more data can be appended to a busy chunk.
fs_path	The path to the file if the chunk is backed by the filesystem.
_head	A mk_list head for linking the chunk into the input instance's list of chunks.

Sources: source/src/flb_input_chunk.c, source/src/flb_dump.c:150-158

The lifecycle of an input chunk is managed by functions that create, append data to, and manage the state of the chunk.

Sources: source/src/flb_input_chunk.c

Storage Backends

Sources: source/src/flb_storage.c, source/lib/rocksdb/README.md

The storage layer is designed to be pluggable, allowing for different backend implementations. The default backend is filesystem, which uses the ChunkIO library to store chunks as files on disk.

The flb_storage_input_create() function is responsible for initializing the storage context for an input plugin instance, selecting the appropriate backend based on configuration.

/* from source/src/flb_storage.c */
struct flb_storage_input *flb_storage_input_create(
    struct flb_input_instance *in, struct flb_config *config)
{
    // ... logic to select and initialize storage backend ...
}

Sources: source/src/flb_storage.c

Besides the default filesystem backend, the agent can be compiled with support for RocksDB, a high-performance embedded key-value store. This provides a more robust and scalable buffering mechanism for demanding environments.

Sources: source/lib/rocksdb/README.md

Monitoring and Diagnostics

Sources: source/src/flb_dump.c

FluentDo Agent provides a dump functionality that outputs the current state of the storage and buffering system to stderr. This is invaluable for debugging and monitoring the agent's health. The flb_dump() function orchestrates this process.

Storage Layer Statistics

The dump_storage() static function provides a high-level summary of the ChunkIO layer.

/* from source/src/flb_dump.c */
static void dump_storage(struct flb_config *ctx)
{
    struct cio_stats storage_st;

    fprintf(stderr, "\n===== Storage Layer =====\n");
    cio_stats_get(ctx->cio, &storage_st);

    fprintf(stderr, "total chunks     : %i\n", storage_st.chunks_total);
    fprintf(stderr, "├─ mem chunks    : %i\n", storage_st.chunks_mem);
    fprintf(stderr, "└─ fs chunks     : %i\n", storage_st.chunks_fs);
    fprintf(stderr, "   ├─ up         : %i\n", storage_st.chunks_fs_up);
    fprintf(stderr, "   └─ down       : %i\n", storage_st.chunks_fs_down);
}

Sources: source/src/flb_dump.c:191-206

This output provides the following metrics:

Metric	Description
total chunks	Total number of chunks managed by ChunkIO.
mem chunks	Number of chunks stored only in memory.
fs chunks	Number of chunks backed by the filesystem.
up	Number of filesystem chunks currently loaded into memory.
down	Number of filesystem chunks currently on disk (not in memory).

Sources: source/src/flb_dump.c:191-206

Input Plugin Details

The dump_input_chunks() static function provides a detailed breakdown for each input plugin instance.

This includes:

• Overlimit Status: Whether the input instance has breached its mem_buf_limit.
• Memory Usage: The current memory size of all chunks associated with the instance.
• Task Summary: The number of total, new, and running tasks, and their combined data size.
• Chunk Summary: The number of up, down, and busy chunks.

Sources: source/src/flb_dump.c:38-188

Conclusion

The Storage and Buffering system is a fundamental feature that underpins the reliability of the FluentDo Agent. By abstracting data handling into chunks managed by the versatile ChunkIO library, it provides a robust mechanism for both memory and filesystem-backed buffering. This architecture ensures that data is not lost during transmission, allows the agent to handle backpressure gracefully, and provides clear diagnostics for monitoring the state of the data pipeline. The pluggable nature of the storage backend, with support for alternatives like RocksDB, further enhances its flexibility for various operational demands.