OpenTelemetry Integration

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/in_opentelemetry/opentelemetry.c
• source/plugins/out_opentelemetry/opentelemetry.c
• source/lib/fluent-otel-proto/README.md
• source/src/opentelemetry/flb_opentelemetry_logs.c
• source/lib/fluent-otel-proto/CMakeLists.txt

FluentDo Agent provides native integration with the OpenTelemetry (OTel) ecosystem through a dedicated input plugin, output plugin, and a core library for handling OTel's protobuf data format. This allows the agent to function as a seamless component in an OpenTelemetry observability pipeline, capable of receiving, processing, and forwarding OTel data.

The integration is built upon the fluent-otel-proto static library, which provides the C-language interfaces for the OpenTelemetry data models for logs, traces, and metrics. The in_opentelemetry plugin acts as an OTLP (OpenTelemetry Protocol) receiver, while the out_opentelemetry plugin forwards data to any OTLP-compatible endpoint.

Core Component: fluent-otel-proto Library

The fluent-otel-proto library is a fundamental component that enables OpenTelemetry integration. It is a static library that contains the C structures and functions corresponding to the OpenTelemetry Protocol (OTLP) data models. It is responsible for the serialization and deserialization of OTel data.

Sources: source/lib/fluent-otel-proto/README.md

Build Process and Dependencies

The library's C source files are generated from the official .proto files from the OpenTelemetry project. This generation process is handled by CMake and relies on two key external repositories:

1. fluent/protobuf-c: A fork of the official protobuf-c library, modified to support options features from proto3, which is required for OTel Metrics.
2. open-telemetry/opentelemetry-proto: The official repository containing the .proto file definitions for OTel signals.

The regeneration of C files is not part of a standard build and must be explicitly enabled.

Sources: source/lib/fluent-otel-proto/README.md, source/lib/fluent-otel-proto/CMakeLists.txt:4-7

The diagram below illustrates the process of generating the C source files for the library.

Supported Signals and Build Options

The library can be configured at build time to include support for different OTel signals (Logs, Traces, Metrics). The following CMake options control which components are generated and included in the library.

Build Option	Description	Default
FLUENT_PROTO_REGENERATE	Enables the regeneration of C source files from .proto files.	No
FLUENT_PROTO_COMMON	Includes the C interface for common.proto.	Yes
FLUENT_PROTO_RESOURCE	Includes the C interface for resource.proto.	Yes
FLUENT_PROTO_TRACE	Includes C interfaces for trace.proto and trace_service.proto.	Yes
FLUENT_PROTO_LOGS	Includes C interfaces for logs.proto and logs_service.proto.	Yes
FLUENT_PROTO_METRICS	Includes C interfaces for metrics.proto and metrics_service.proto.	Yes

Sources: source/lib/fluent-otel-proto/CMakeLists.txt:8-14, source/lib/fluent-otel-proto/README.md

Input Plugin: in_opentelemetry

The in_opentelemetry plugin functions as an OTLP/HTTP receiver, allowing FluentDo Agent to ingest logs, traces, and metrics from OpenTelemetry-instrumented applications or collectors. It exposes HTTP endpoints for each signal type.

Sources: source/plugins/in_opentelemetry/opentelemetry.c

Data Ingestion Flow

The plugin sets up an HTTP server that listens for incoming OTLP/HTTP requests. When a request is received, it is processed based on its URI (/v1/logs, /v1/traces, /v1/metrics). The Protobuf payload is then decoded and converted into Fluent Bit's internal msgpack format, which is then injected into the processing pipeline.

Configuration

The behavior of the in_opentelemetry plugin can be customized with the following configuration options.

Property	Description	Default
tag	Sets the tag for incoming records.	otlp
add_http_headers	If enabled, adds HTTP headers from the request to the log record.	false
raw_traces	If enabled, forwards traces without decoding them.	false

Sources: source/plugins/in_opentelemetry/opentelemetry.c:359-373

Output Plugin: out_opentelemetry

The out_opentelemetry plugin sends Fluent Bit records to an OTLP/HTTP-compatible endpoint. It handles the conversion of records from Fluent Bit's internal format to the OTLP format and manages the network communication.

Sources: source/plugins/out_opentelemetry/opentelemetry.c

Data Egress Flow

The plugin takes batches of records from the Fluent Bit engine, encodes them into the OTLP log format, and sends them as an HTTP POST request to the configured downstream endpoint.

Configuration

The out_opentelemetry plugin is configured through the standard output plugin settings. Key properties include:

Property	Description	Default
logs_uri	The URI for the OTLP logs endpoint.	/v1/logs
traces_uri	The URI for the OTLP traces endpoint.	/v1/traces
metrics_uri	The URI for the OTLP metrics endpoint.	/v1/metrics
header	Specify a custom HTTP header in Key Value format. Can be used multiple times.	NULL
http_user	Basic authentication username.	NULL
http_passwd	Basic authentication password.	NULL
tls	Enable or disable TLS.	false
tls.verify	Enable or disable certificate validation.	true
tls.ca_file	Path to a CA certificate file.	NULL
tls.crt_file	Path to a client certificate file.	NULL
tls.key_file	Path to a client private key file.	NULL

Sources: source/plugins/out_opentelemetry/opentelemetry.c:138-160, source/plugins/out_opentelemetry/opentelemetry.c:395-460

Data Transformation

A crucial part of the OpenTelemetry integration is the transformation between Fluent Bit's native msgpack data format and the structured OTLP format. This logic is primarily encapsulated in flb_opentelemetry_logs.c.

Sources: source/src/opentelemetry/flb_opentelemetry_logs.c

Fluent Bit Record to OTLP LogRecord Mapping

When the out_opentelemetry plugin processes a log, it invokes flb_opentelemetry_logs_encode_records to perform the conversion. The function iterates through each msgpack record and maps its fields to the corresponding fields in an OTLP LogRecord.

The following sequence diagram shows the mapping process for a single record.

Key field mappings include:

• Timestamp: The record's timestamp is mapped to observed_time_unix_nano.
• Body: The value of the log or message key in the record is used for the body of the LogRecord.
• Severity: The severity_text and severity_number fields are populated from corresponding keys in the record.
• Trace Context: trace_id and span_id are extracted if present.
• Attributes: All other key-value pairs in the msgpack record are converted into KeyValue pairs in the attributes field of the LogRecord.

Sources: source/src/opentelemetry/flb_opentelemetry_logs.c:66-189

Conclusion

The OpenTelemetry integration in FluentDo Agent is comprehensive, providing both ingestion and egress capabilities for OTLP data. By leveraging the fluent-otel-proto library, the in_opentelemetry and out_opentelemetry plugins enable the agent to act as a powerful and flexible log processor within a modern, vendor-neutral observability stack. The clear separation of concerns between protocol handling, data transformation, and plugin logic creates a robust and maintainable implementation.