Input Plugins Overview

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/CMakeLists.txt
• source/src/fluent-bit.c
• README.md
• source/lib/rocksdb/plugin/README.md
• source/include/fluent-bit/flb_input.h
• source/include/fluent-bit/flb_config.h

Input plugins are the primary mechanism for data collection in FluentDo Agent. They are responsible for gathering logs, metrics, and other telemetry data from a wide variety of sources, such as files, network sockets, system metrics, and specific application endpoints. The agent's architecture is built around these plugins, allowing for a flexible and extensible system where new data sources can be integrated by adding new input plugins.

FluentDo Agent, being a hardened distribution of Fluent Bit, includes a curated set of production-essential plugins. These plugins are compiled into the agent statically, and their behavior is configured through the main configuration files or command-line arguments. The system also supports loading external plugins dynamically.

Sources: README.md, source/src/fluent-bit.c:168-176, source/plugins/CMakeLists.txt

Plugin Registration and Build Process

Input plugins are registered and compiled into the agent using a CMake-based build system. The core of this process is the REGISTER_IN_PLUGIN macro defined in the main CMakeLists.txt for plugins.

Sources: source/plugins/CMakeLists.txt:39-68

The REGISTER_IN_PLUGIN Macro

This CMake macro handles the logic for adding an input plugin to the build. It performs the following steps:

1. Declares the external plugin structure (e.g., extern struct flb_input_plugin in_tail_plugin;).
2. Generates C code to allocate memory for the plugin structure and add it to the configuration's list of available input plugins (config->in_plugins).
3. Adds the plugin's source directory to the build process.
4. Adds the compiled plugin library (flb-plugin-in_tail) to the list of plugins to be linked.

This process is driven by the configure_file command in CMake, which generates the flb_plugins.h header file from flb_plugins.h.in. This header contains the necessary declarations and registration code.

Sources: source/plugins/CMakeLists.txt:39-68, source/plugins/CMakeLists.txt:435-438

The following diagram illustrates the static plugin registration flow during the build process.

External Plugins

In addition to statically compiled plugins, FluentDo Agent can load external plugins from shared library files (.so or .dll). This is handled using the -e or --plugin=FILE command-line option. The agent also supports specifying external plugins via the EXT_IN_PLUGINS CMake variable, which allows them to be registered similarly to internal plugins.

Sources: source/src/fluent-bit.c:686-692, source/plugins/CMakeLists.txt:414-419

Core Plugin Structure

The behavior and lifecycle of every input plugin are defined by a set of standardized C structures. The two primary structures are flb_input_plugin and flb_input_instance.

Sources: source/include/fluent-bit/flb_input.h

flb_input_plugin

This structure defines the plugin itself. It contains metadata like the plugin's name and description, along with function pointers that implement the plugin's lifecycle callbacks (e.g., initialization, collection, exit). A single flb_input_plugin is registered for each type of input (e.g., one for tail, one for cpu).

flb_input_instance

This structure represents a configured instance of a plugin. When a user defines an [INPUT] section in the configuration, the agent creates an flb_input_instance. This instance holds the specific configuration for that block, such as the tag, the path to a file, or a network port, and maintains its own operational state. Multiple instances of the same plugin can exist with different configurations.

The relationship between the core configuration, plugin definitions, and running instances is shown below.

Configuration

Input plugins can be configured through the main configuration file or via command-line arguments. The command-line interface provides a quick way to define a simple pipeline.

Sources: source/src/fluent-bit.c:699-722

Command-Line Options

The following command-line options are used to configure input plugins:

Option	Long Option	Description
-i	--input=INPUT	Specifies an input plugin to use (e.g., tail). This creates a new input instance.
-p	--prop="A=B"	Sets a configuration property for the most recently defined plugin instance.
-t	--tag=TAG	Sets the tag for the most recently defined plugin instance. Equivalent to -p tag=....
-m	--match=MATCH	Sets the match property for filter or output plugins.

Sources: source/src/fluent-bit.c:120,123,124,126

When these options are used, the fluent-bit.c main function parses them and dynamically constructs configuration sections, which are then used to initialize the plugin instances.

The following diagram shows the sequence for creating and configuring an input plugin instance via command-line arguments.

Available Input Plugins

The source/plugins/CMakeLists.txt file lists all the input plugins that are registered to be built with the agent. This provides a definitive list of supported data sources.

Plugin Name	Description
in_fluentdo	FluentDo custom input plugin.
in_blob	Generic blob input.
in_cpu	Gathers CPU usage metrics (Linux only).
in_mem	Gathers memory usage metrics (Linux only).
in_thermal	Gathers thermal zone metrics (Linux only).
in_kmsg	Reads messages from the Kernel log buffer (Linux only).
in_proc	Gathers process metrics (Linux only).
in_disk	Gathers disk I/O metrics (Linux only).
in_systemd	Reads logs from Systemd/Journald (Linux only).
in_netif	Gathers network interface metrics (Linux only).
in_docker	Gathers Docker container metrics (Linux only).
in_docker_events	Gathers Docker events (Linux only).
in_podman_metrics	Gathers Podman metrics (Linux only).
in_process_exporter_metrics	Gathers metrics from a process exporter.
in_ebpf	Collects data using eBPF (Linux only).
in_gpu_metrics	Collects GPU metrics (Linux only).
in_node_exporter_metrics	Scrapes metrics from a Node Exporter endpoint.
in_kubernetes_events	Collects Kubernetes events.
in_kafka	Consumes messages from Apache Kafka topics.
in_fluentbit_metrics	Gathers internal Fluent Bit metrics.
in_git_config	Reads configuration from a Git repository.
in_prometheus_scrape	Scrapes metrics from Prometheus endpoints.
in_prometheus_textfile	Reads metrics from Prometheus textfile format.
in_emitter	Listens for records from other Fluent Bit instances.
in_tail	Tails files, the most common log collection method.
in_dummy	Generates dummy log events.
in_head	Reads a fixed number of lines from the head of a file.
in_health	Gathers health check metrics.
in_http	Listens for data over an HTTP endpoint.
in_collectd	Listens for metrics from Collectd.
in_statsd	Listens for metrics from a StatsD server.
in_opentelemetry	Receives traces, metrics, and logs via OpenTelemetry Protocol (OTLP).
in_elasticsearch	Queries and ingests data from Elasticsearch.
in_splunk	Receives data from Splunk forwarders.
in_prometheus_remote_write	Receives metrics via Prometheus remote write protocol.
in_event_test	Test plugin for event loop messaging.
in_event_type	Test plugin for sending different event types.
in_storage_backlog	Reads data from storage backlog.
in_nginx_exporter_metrics	Scrapes metrics from an Nginx exporter.
in_stream_processor	Input for the stream processor engine.
in_winlog	Reads from the Windows Event Log (Windows only).
in_winstat	Gathers performance counters (Windows only).
in_winevtlog	Reads from the Windows Event Log XML (Windows only).
in_windows_exporter_metrics	Scrapes metrics from a Windows exporter (Windows only).
in_serial	Reads data from a serial port.
in_stdin	Reads data from standard input.
in_syslog	Listens for syslog messages.
in_exec	Executes an external program and reads its output.
in_udp	Listens for data over UDP.
in_tcp	Listens for data over TCP.
in_unix_socket	Listens for data on a Unix socket.
in_mqtt	Subscribes to an MQTT topic.
in_lib	For library mode usage.
in_forward	Listens for data using the Fluentd forward protocol.
in_random	Generates random data.

Sources: source/plugins/CMakeLists.txt:248-360

Summary

Input plugins are a cornerstone of FluentDo Agent's data collection capabilities. They are discovered and loaded at build time through a CMake-based registration system, with support for dynamic loading of external plugins. Each plugin is defined by a standard flb_input_plugin structure, and configured instances are managed as flb_input_instance structures. This modular design allows the agent to ingest data from a vast and diverse set of sources, making it a highly versatile tool for modern observability pipelines.