Output Plugins Overview

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/CMakeLists.txt
• source/src/fluent-bit.c
• README.md
• source/lib/rocksdb/plugin/README.md
• source/lib/monkey/CMakeLists.txt

FluentDo Agent, a hardened distribution of Fluent Bit, utilizes a flexible plugin-based architecture to process and route data. Output plugins are a critical component of this architecture, responsible for sending processed logs, metrics, and traces to a wide variety of destinations. They are the final step in the data pipeline, handling the delivery of data to services like Elasticsearch, Kafka, Splunk, and various cloud provider endpoints.

This document provides an overview of how output plugins are defined, registered, and configured within the FluentDo Agent build system and runtime environment.

Sources: README.md, source/src/fluent-bit.c:116-121

Plugin Registration

Output plugins are registered within the build system using a CMake macro called REGISTER_OUT_PLUGIN. This macro handles the necessary steps to include a plugin in the final binary.

Sources: source/plugins/CMakeLists.txt:82-127

REGISTER_OUT_PLUGIN Macro

The REGISTER_OUT_PLUGIN macro is defined in source/plugins/CMakeLists.txt. Its primary role is to conditionally add a plugin's source code to the build, declare it for runtime initialization, and link it.

The registration process can be visualized as follows:

This diagram shows the logic flow of the REGISTER_OUT_PLUGIN macro in CMake.

Sources: source/plugins/CMakeLists.txt:82-127

The macro performs these key actions:

1. Checks for a flag: It checks if a corresponding FLB_<NAME> flag is set or if an external path is provided.
2. Declares the plugin struct: It adds a C extern declaration for the plugin's flb_output_plugin struct.
3. Generates registration code: It creates C code that will be compiled into the core. This code allocates memory for the plugin struct, copies the plugin definition, and adds it to the main configuration's out_plugins list.
4. Adds Subdirectory: It calls add_subdirectory() on the plugin's source directory, triggering its own CMakeLists.txt to be processed.
5. Adds to Plugin List: It appends the plugin library name (e.g., flb-plugin-out_http) to a global list of plugins to be linked.

A special case exists for plugins written in Zig, which involves a different C code generation path to initialize the plugin via flb_zig_native_output_plugin_init.

Sources: source/plugins/CMakeLists.txt:99-113

External Plugin Registration

The system also supports registering external output plugins at build time via the EXT_OUT_PLUGINS CMake variable. This variable can contain a comma-separated list of plugins to include. The build script iterates through this list and calls REGISTER_OUT_PLUGIN for each one.

Sources: source/plugins/CMakeLists.txt:444-449

Available Output Plugins

The source/plugins/CMakeLists.txt file contains a comprehensive list of output plugins that are registered by default.

Plugin Name	Description
out_azure	Microsoft Azure Blob Storage.
out_azure_blob	Microsoft Azure Blob Storage.
out_azure_logs_ingestion	Microsoft Azure Logs Ingestion API.
out_azure_kusto	Microsoft Azure Data Explorer (Kusto).
out_bigquery	Google BigQuery.
out_counter	Counts the number of records.
out_datadog	Datadog.
out_es	Elasticsearch.
out_exit	Exits after processing a certain number of records.
out_file	Writes records to a file.
out_forward	Forwards records to another Fluent Bit or Fluentd instance.
out_http	Sends records over HTTP.
out_influxdb	InfluxDB Time Series Database.
out_logdna	LogDNA.
out_loki	Grafana Loki.
out_kafka	Apache Kafka (if FLB_KAFKA is enabled).
out_kafka_rest	Kafka REST Proxy.
out_nats	NATS.
out_nrlogs	New Relic Logs.
out_null	Discards all records.
out_opensearch	OpenSearch.
out_oracle_log_analytics	Oracle Cloud Infrastructure Logging Analytics.
out_plot	Generates a file for plotting (not for Windows).
out_pgsql	PostgreSQL.
out_retry	A wrapper to retry sending data with other output plugins.
out_skywalking	Apache SkyWalking.
out_slack	Sends notifications to Slack.
out_splunk	Splunk.
out_stackdriver	Google Cloud Logging (Stackdriver).
out_stdout	Prints records to standard output.
out_syslog	Sends records to a Syslog server.
out_tcp	Sends records over a raw TCP socket.
out_udp	Sends records over a raw UDP socket.
out_lib	For library mode usage.
out_flowcounter	Counts records and bytes per second.
out_gelf	Graylog Extended Log Format (GELF).
out_websocket	Sends records over WebSocket.
out_cloudwatch_logs	AWS CloudWatch Logs.
out_kinesis_firehose	AWS Kinesis Data Firehose.
out_kinesis_streams	AWS Kinesis Data Streams.
out_opentelemetry	OpenTelemetry Protocol (OTLP).
out_prometheus_exporter	Exposes metrics in Prometheus format.
out_prometheus_remote_write	Sends metrics to a Prometheus remote_write endpoint.
out_s3	AWS S3.
out_chronicle	Google Chronicle.
out_zig_demo	A demo plugin written in Zig (if FLB_ZIG is enabled).

Sources: source/plugins/CMakeLists.txt:301-365

Runtime Configuration

Output plugins are configured at runtime, primarily through command-line arguments or a configuration file.

Command-Line Options

The main executable provides several options to configure output plugins directly from the command line.

Option	Long Option	Description
-o	--output=OUTPUT	Specifies an output plugin to use.
-m	--match=MATCH	Sets a match pattern for the preceding output plugin.
-p	--prop="A=B"	Sets a configuration property for the preceding plugin.
-t	--tag=TAG	A shortcut for -p tag=TAG.
-e	--plugin=FILE	Loads an external plugin from a shared library.

Sources: source/src/fluent-bit.c:100-115

The following diagram illustrates the sequence of configuring and initializing an output plugin via command-line flags.

This sequence diagram shows how command-line arguments are parsed to configure and initialize an output plugin.

Sources: source/src/fluent-bit.c:541-650

Conclusion

The output plugin system in FluentDo Agent is a robust and extensible mechanism for data egress. Through a combination of a CMake-based build system for static registration and the ability to load external shared libraries at runtime, it offers significant flexibility. The REGISTER_OUT_PLUGIN macro is the cornerstone of the build-time registration, while a clear set of command-line arguments and configuration file options provide powerful runtime control over data routing and formatting. This architecture allows the agent to support a vast ecosystem of destinations, making it a versatile tool for modern observability pipelines.

Sources: source/plugins/CMakeLists.txt, source/src/fluent-bit.c