Service & Pipeline Configuration
Relevant source files
The following files were used as context for generating this wiki page:
FluentDo Agent, a hardened distribution of Fluent Bit, uses a flexible configuration system to define its behavior and data processing pipelines. The configuration is managed through files that specify global service settings and a series of pipeline stages: inputs, filters, and outputs. This system allows for the collection, transformation, and routing of data from various sources to multiple destinations.
The core of the configuration is the data pipeline, which dictates how logs, metrics, and traces are ingested, processed, and delivered. Each stage of the pipeline is defined in a dedicated configuration block, and data is routed between stages using a tag-and-match system. Global agent settings, such as flushing intervals, logging verbosity, and buffering mechanisms, are controlled in a main [SERVICE] block.
Core Concepts: The Data Pipeline
The FluentDo Agent processes data through a pipeline composed of three main plugin types: Input, Filter, and Output.
- Input: Collects data from sources. Each input plugin assigns a "tag" to the data it ingests.
- Filter: Modifies, enriches, or drops data as it flows through the pipeline. Filters select data based on a "match" rule against the data's tag.
- Output: Sends data to a final destination. Outputs also use a "match" rule to select which data to send.
The diagram below illustrates the basic data flow through the agent.
This diagram shows the fundamental data flow from a source, through the Input, Filter, and Output plugins, to a destination.
Sources: source/src/flb_input.c, source/src/flb_filter.c, source/src/flb_output.c
[SERVICE] Block: Global Configuration
The [SERVICE] block defines the global operational parameters for the FluentDo Agent. These settings control the core engine's behavior, including data flushing, logging, and storage.
Sources: source/conf/fluent-bit.conf:1-55, source/src/flb_sosreport.c:198-203
| Property | Description | Default Value (from conf) |
|---|---|---|
flush | The interval in seconds to flush records to a destination. | 1 |
daemon | Runs the agent in the background (On) or foreground (Off). | Off |
log_level | Sets the verbosity level. Can be error, warning, info, debug, or trace. | info |
parsers_file | Specifies an optional 'Parsers' configuration file. | parsers.conf |
plugins_file | Specifies an optional 'Plugins' configuration file for external plugins. | plugins.conf |
http_server | Enables or disables the built-in HTTP server for metrics. | Off |
http_listen | The IP address for the HTTP server to listen on. | 0.0.0.0 |
http_port | The port for the HTTP server. | 2020 |
Storage and Buffering
The agent provides memory and filesystem buffering to handle backpressure and ensure data reliability. These features are configured within the [SERVICE] block.
Sources: source/conf/fluent-bit.conf:31-55
| Property | Description |
|---|---|
storage.metrics | Publishes storage pipeline metrics at the /api/v1/storage endpoint if http_server is enabled. |
storage.path | An absolute path to a directory for filesystem data buffers (chunks). |
storage.sync | Sets the filesystem synchronization mode. Can be normal or full. |
storage.checksum | Enables CRC32 data integrity checks for reading and writing filesystem chunks. |
storage.backlog.mem_limit | Sets a memory limit for processing backlog data from the storage path. |
Pipeline Blocks: [INPUT], [FILTER], [OUTPUT]
The data pipeline is constructed using [INPUT], [FILTER], and [OUTPUT] configuration blocks. Each block defines an instance of a specific plugin and its configuration.
[INPUT] - Data Sources
Input plugins are the data sources for the pipeline. Each [INPUT] block defines an instance of an input plugin.
Key properties for [INPUT] blocks:
Name: The name of the input plugin to use (e.g.,cpu,dummy,winlog).Tag: A string tag assigned to all records generated by this input. This tag is used for routing.
Sources: source/src/flb_input.c, source/src/flb_sosreport.c:206-235
Example: CPU Metrics Input
This example configures the cpu input plugin to collect CPU metrics every second and tag them with cpu.local.
[INPUT]
name cpu
tag cpu.local
# Read interval (sec) Default: 1
interval_sec 1
Sources: source/conf/fluent-bit.conf:57-63
Example: Windows Event Log Input
This example configures the winlog input to collect events from the 'Setup' and 'Windows PowerShell' channels.
[INPUT]
Name winlog
Channels Setup,Windows PowerShell
Interval_Sec 1
Sources: source/conf/fluent-bit-win32.conf:57-61
[FILTER] - Data Transformation
Filter plugins modify records as they pass through the pipeline. They can be used to enrich data with metadata, drop unwanted records, or restructure log content.
Key properties for [FILTER] blocks:
Name: The name of the filter plugin.Match: A wildcard pattern that is matched against record tags to determine if the filter should be applied.
Sources: source/src/flb_filter.c, source/src/flb_sosreport.c:238-246
[OUTPUT] - Data Destinations
Output plugins send records to a final destination, such as a database, a cloud service, or standard output.
Key properties for [OUTPUT] blocks:
Name: The name of the output plugin (e.g.,stdout).Match: A wildcard pattern that is matched against record tags to select which records to send to this output.
Sources: source/src/flb_output.c, source/src/flb_sosreport.c:249-281
Example: Standard Output
This example configures the stdout output plugin to print all records (match *) to the console.
[OUTPUT]
name stdout
match *
Sources: source/conf/fluent-bit.conf:65-67
Data Routing: Tags and Matching
Routing is the mechanism that directs data from inputs to the correct filters and outputs. It is based on a simple yet powerful system of tags and match patterns.
- Tagging: Every record entering the pipeline from an
[INPUT]plugin is assigned aTag. - Matching:
[FILTER]and[OUTPUT]plugins use aMatchorMatch_Regexproperty to select records. TheMatchproperty supports wildcards (*) to match multiple tags.
The flb_router is the core component responsible for this logic, ensuring that a record from an input instance is passed to all filter and output instances whose Match pattern corresponds to the record's tag.
Sources: source/src/flb_router.c
The following diagram shows a more complex routing scenario with multiple inputs and outputs.
This diagram illustrates how tags and match patterns route data. app.logs data goes through Filter 1 and then to Output X and Output Z. metrics.cpu data goes directly to Output Y and Output Z.
Sources: source/src/flb_router.c, source/src/flb_sosreport.c:243,253
Summary
The FluentDo Agent's configuration is built around the concepts of a central [SERVICE] block for global settings and a data pipeline defined by [INPUT], [FILTER], and [OUTPUT] blocks. This structure provides a declarative and powerful way to control data flow. The tag-and-match system is the cornerstone of its routing logic, enabling flexible pipelines that can handle diverse data sources and destinations within a single agent instance. Understanding these core components is essential for effectively deploying and managing the FluentDo Agent.