Configuration File Structure (YAML)
Relevant source files
The following files were used as context for generating this wiki page:
FluentDo Agent utilizes a YAML-based configuration format for defining its behavior, including data pipelines, service settings, and dynamic updates. This structure provides a human-readable and flexible way to manage the agent's operations. The default configuration file is fluent-bit.yaml, which serves as the primary entry point for the agent's settings.
The configuration system is designed to be robust, supporting dynamic includes and atomic hot-reloading, particularly through specialized plugins like in_git_config. The core parsing is handled by the yyjson library, which processes the YAML file into an internal configuration context that the agent uses at runtime.
Sources: README.md, config/fluent-bit.yaml, source/src/config_format/flb_cf_yaml.c, source/plugins/in_git_config/git_config.c
Core Concepts
The YAML configuration is organized into several top-level sections. The primary sections are service and pipeline, which are fundamental to Fluent Bit's operation. The system also supports includes for modularity and a special customs section for preserving local settings during remote updates.
service:
# Service-level configuration
...
pipeline:
# Pipeline definition (inputs, filters, outputs)
...
includes:
# List of other configuration files to include
...
customs:
# Custom definitions preserved by in_git_config
...
Sources: config/fluent-bit.yaml, source/plugins/in_git_config/git_config.c:221-252
YAML Parsing and Processing
The agent processes YAML configuration files by reading the file and converting it into an internal, format-agnostic representation. This process is primarily managed by the functions within flb_cf_yaml.c.
The function flb_cf_yaml_create is the entry point for parsing a YAML file. It uses the yyjson library, a high-performance C JSON/YAML parser, to read the file content. The yyjson_read_file function is called to parse the file, with flags that allow for non-standard features like comments and trailing commas, enhancing user convenience. Once parsed, the YAML document is traversed, and its contents are used to populate the flb_config structure.
Sources: source/src/config_format/flb_cf_yaml.c, source/lib/yyjson-0.12.0/doc/doxygen/html/index.html
The following diagram illustrates the YAML file parsing flow.
Main Configuration Sections
service
The service block defines global properties for the FluentDo Agent. These settings control the core behavior of the agent, such as flushing intervals, logging, and its HTTP server.
Sources: config/fluent-bit.yaml:2-10
| Property | Description | Default Value (from fluent-bit.yaml) |
|---|---|---|
flush | The interval in seconds to flush data from input plugins to output plugins. | 5 |
log_level | The logging verbosity level. | info |
daemon | If true, the agent runs in the background. | false |
parsers_file | Path to the parsers configuration file. | parsers.conf |
http_server | If true, enables the built-in HTTP server for monitoring. | true |
http_listen | The IP address for the HTTP server to listen on. | 0.0.0.0 |
http_port | The port for the HTTP server. | 2020 |
storage.path | Path for the filesystem buffering mechanism. | /var/log/flb-storage/ |
Sources: config/fluent-bit.yaml:2-10
pipeline
The pipeline block is where the data processing flow is defined. It is a container for input, filter, and output plugin configurations. Each plugin is defined with a name and other specific properties.
A simple pipeline example:
pipeline:
inputs:
- name: tail
path: /var/log/containers/*.log
filters:
- name: kubernetes
match: kube.*
outputs:
- name: stdout
match: '*'
This configuration defines a pipeline that tails log files, enriches them with Kubernetes metadata, and prints the results to standard output.
Sources: config/fluent-bit.yaml:12-38
Advanced Configuration Management (in_git_config)
The in_git_config input plugin provides a powerful mechanism for managing agent configuration dynamically from a Git repository. It polls a remote repository for changes and triggers a hot-reload of the agent when a new configuration is detected.
Sources: source/plugins/in_git_config/git_config.c
The includes Directive
To maintain separation between remotely managed configuration and local customizations, the in_git_config plugin uses an includes directive. When a new configuration is fetched from Git, the plugin prepends an includes block to it, pointing to a local header.yaml file. This ensures that local settings are always loaded first.
The create_config_with_include function is responsible for this behavior.
/* source/plugins/in_git_config/git_config.c:309-313 */
/* Prepend include directive */
if (flb_sds_printf(&config_with_include, "includes:\n - %s\n\n", header_path) == NULL) {
goto cleanup;
}
Sources: source/plugins/in_git_config/git_config.c:309-313
The customs Section
The customs section is a special, user-defined block in the main local configuration file. The in_git_config plugin extracts this section and saves it into header.yaml. This mechanism allows site-specific configurations (like custom parsers or environment variables) to be preserved and applied on top of any configuration pulled from the remote Git repository.
The extract_customs_section function scans the YAML file for a top-level customs: key and extracts its entire content.
Sources: source/plugins/in_git_config/git_config.c:221-252
Configuration Lifecycle (Staging, Commit, Rollback)
The in_git_config plugin implements a safe, atomic update process using a set of reference files (.ref) to manage different versions of the configuration. This prevents the agent from being left in a broken state due to a faulty update.
| Ref File | Description |
|---|---|
new.ref | Points to the new configuration file that has been downloaded and is staged for hot-reloading. |
cur.ref | Points to the currently active and validated configuration file. |
old.ref | Points to the previous active configuration, serving as a backup for rollbacks. |
Sources: source/plugins/in_git_config/git_config.c:45-120
The lifecycle of a configuration update is as follows:
This flow is managed by functions like config_add (staging), config_commit (promotion), and config_rollback (failure recovery).
Sources: source/plugins/in_git_config/git_config.c:332-493
Conclusion
The YAML configuration system in FluentDo Agent provides a structured and extensible method for defining agent behavior. The base service and pipeline sections offer standard Fluent Bit functionality, while advanced features like the in_git_config plugin enable robust, dynamic configuration management suitable for large-scale, automated environments. The use of includes and the special customs section ensures a clean separation between remote, centrally managed configurations and local, node-specific customizations.