Filter Plugins Overview

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/CMakeLists.txt
• source/src/fluent-bit.c
• README.md
• source/lib/librdkafka-2.10.1/CONTRIBUTING.md
• source/lib/monkey/CMakeLists.txt

Filter plugins are a core component of the FluentDo Agent data processing pipeline. They provide the ability to modify, enrich, or drop incoming log records before they are sent to an output destination. Filters are positioned between input and output plugins, allowing for powerful in-flight data manipulation, such as parsing nested JSON, adding Kubernetes metadata, or filtering out unwanted records.

The agent includes a variety of built-in filter plugins and supports the registration of external plugins, making the system highly extensible. This document provides an overview of the filter plugin architecture, registration process, and configuration within the FluentDo Agent.

Plugin Registration and Build Process

Filter plugins are registered and included in the build process through a CMake-based system. The primary mechanism for this is the REGISTER_FILTER_PLUGIN macro defined in the main plugins CMakeLists.txt file.

Sources: source/plugins/CMakeLists.txt:203-228

REGISTER_FILTER_PLUGIN Macro

This CMake macro handles the logic for registering a filter plugin, generating the necessary C code for its initialization, and including it in the build.

The following diagram illustrates the logic of the REGISTER_FILTER_PLUGIN macro.

This diagram shows the conditional build process for a filter plugin based on its corresponding CMake flag.

Sources: source/plugins/CMakeLists.txt:203-228

The macro performs the following steps:

1. It checks if a build option FLB_<NAME> (where <NAME> is the uppercased plugin name) is enabled. This allows for conditional compilation of plugins.
2. If the plugin is enabled, it appends a C extern declaration for the plugin's definition struct (extern struct flb_filter_plugin ${p_name}_plugin;) to a variable that will be used to generate flb_plugins.h.
3. It generates C code that, at runtime, will:
   • Allocate memory for a flb_filter_plugin struct.
   • Copy the static plugin definition into the newly allocated memory.
   • Add the new plugin instance to the main configuration's filter_plugins linked list.
4. It calls add_subdirectory() to compile the plugin's source code.
5. It adds the compiled static library (flb-plugin-${p_name}) to a list of plugins to be linked into the main agent binary.

Sources: source/plugins/CMakeLists.txt:203-228

Here is the definition of the macro:

# REGISTER_FILTER_PLUGIN
macro(REGISTER_FILTER_PLUGIN name)
  string(FIND ${name} "=" pos)
  if(pos GREATER -1)
    string(REPLACE "=" ";" list ${name})
    list(GET list 0 p_name)
    list(GET list 1 p_path)
    message(STATUS "EXTERNAL FILTER PLUGIN name='${p_name}' path='${p_path}'")
  else()
    set(p_name ${name})
  endif()

  string(TOUPPER ${p_name} NAME)
  if(FLB_${NAME} OR p_path)
    set(FLB_FILTER_PLUGINS_DECL "${FLB_FILTER_PLUGINS_DECL}extern struct flb_filter_plugin ${p_name}_plugin;\n")

    # C code
    set(C_CODE          "    filter = flb_malloc(sizeof(struct flb_filter_plugin));\n")
    set(C_CODE "${C_CODE}    if (!filter) {\n")
    set(C_CODE "${C_CODE}        flb_errno();\n")
    set(C_CODE "${C_CODE}        return -1;\n")
    set(C_CODE "${C_CODE}    }\n")
    set(C_CODE "${C_CODE}    memcpy(filter, &${p_name}_plugin, sizeof(struct flb_filter_plugin));\n")
    set(C_CODE "${C_CODE}    mk_list_add(&filter->_head, &config->filter_plugins);\n\n")

    set(FLB_FILTER_PLUGINS_ADD "${FLB_FILTER_PLUGINS_ADD}${C_CODE}")
    if (p_path)
      add_subdirectory(${p_path} ${p_path})
    else()
      add_subdirectory(${p_name})
    endif()
    set(flb_plugins "${flb_plugins}flb-plugin-${p_name};")
  endif()
endmacro()

Sources: source/plugins/CMakeLists.txt:203-228

External Filter Plugins

The build system also supports registering external filter plugins via the EXT_FILTER_PLUGINS CMake variable. When this variable is set, the REGISTER_FILTER_PLUGIN macro is called for each plugin in the list, allowing for out-of-tree plugin development.

Sources: source/plugins/CMakeLists.txt:449-454

Plugin Lifecycle and Instantiation

At runtime, filter plugins are instantiated during the agent's startup phase. The C code generated by the REGISTER_FILTER_PLUGIN macro is responsible for this process.

The following sequence diagram shows how a filter plugin is registered and added to the configuration when the agent starts.

This diagram illustrates the dynamic allocation and registration of a filter plugin into the agent's central configuration.

Sources: source/plugins/CMakeLists.txt:216-221

Configuration

Filter plugins can be configured through the command line or a configuration file.

Command-Line Configuration

The agent provides command-line options to define and configure filter plugins.

• -F, --filter=FILTER: Specifies a filter plugin to use. This creates a [FILTER] section in the configuration.
• -p, --prop="A=B": Sets a configuration property for the most recently defined plugin.
• -m, --match=MATCH: Sets the match property for a filter or output plugin.

When the -F option is used, a new [FILTER] section is created in the configuration context, and its name property is set to the specified plugin name. Subsequent -p options will add properties to this section.

// from source/src/fluent-bit.c
        case 'F':
            s = flb_cf_section_create(cf_opts, "filter", 0);
            if (!s) {
                flb_utils_error(FLB_ERR_FILTER_INVALID);
            }
            flb_cf_section_property_add(cf_opts, s->properties, "name", 0, optarg, 0);
            last_plugin = PLUGIN_FILTER;
            break;

Sources: source/src/fluent-bit.c:645-653

Plugin Help

The agent can display help information for a specific plugin, including its configuration options. This is triggered by using the -h or --help flag after specifying a plugin.

// from source/src/fluent-bit.c
        case 'h':
            if (last_plugin == -1) {
                flb_help(EXIT_SUCCESS, config);
            }
            else {
                flb_help_plugin(EXIT_SUCCESS, FLB_HELP_TEXT,
                                config,
                                last_plugin, cf_opts, s);
            }
            break;

Sources: source/src/fluent-bit.c:687-696

Available Filter Plugins

The FluentDo Agent includes several built-in filter plugins. Their inclusion in the final binary can be controlled by CMake flags.

Plugin Name	Description (Inferred)	Build Condition / Dependency
filter_alter_size	Alters the size of records.	Always built.
filter_aws	Enriches records with AWS metadata.	Always built.
filter_checklist	Checks records against a list of conditions.	Always built.
filter_ecs	Enriches records with ECS metadata.	Always built.
filter_record_modifier	Modifies records based on a set of rules.	Always built.
filter_sysinfo	Adds system information to records.	Always built.
filter_throttle	Throttles the rate of records passing through.	Always built.
filter_throttle_size	Throttles records based on size.	Always built.
filter_tensorflow	Integrates with TensorFlow for data processing.	Always built.
filter_type_converter	Converts data types of fields in a record.	Always built.
filter_kubernetes	Enriches records with Kubernetes metadata.	FLB_REGEX
filter_modify	Modifies records using rules and conditions.	FLB_REGEX
filter_multiline	Merges multiple log lines into a single record.	FLB_REGEX
filter_nest	Nests record fields under a new key.	FLB_REGEX
filter_parser	Applies a parser to a field in a record.	FLB_REGEX
filter_expect	Expects certain fields or values in a record.	FLB_RECORD_ACCESSOR
filter_grep	Filters records based on regular expression matching.	FLB_RECORD_ACCESSOR
filter_rewrite_tag	Rewrites the tag of a record.	FLB_RECORD_ACCESSOR
filter_log_to_metrics	Converts log records to metrics.	FLB_METRICS
filter_lua	Allows record modification using a Lua script.	FLB_LUAJIT
filter_stdout	Prints records to standard output (for debugging).	Always built.
filter_geoip2	Enriches records with GeoIP2 data.	Always built.
filter_nightfall	Integrates with Nightfall DLP for data scanning.	Always built.

Sources: source/plugins/CMakeLists.txt:413-447

Summary

Filter plugins are a fundamental and powerful feature of the FluentDo Agent, enabling flexible and dynamic data processing. The build system, centered around the REGISTER_FILTER_PLUGIN CMake macro, provides a straightforward and extensible way to manage both built-in and external plugins. Configuration is handled through a unified system that supports both command-line arguments and configuration files, giving users fine-grained control over the data pipeline.