Monitoring and HTTP API

Relevant source files

The following files were used as context for generating this wiki page:

• source/src/http_server/api/v1/metrics.c
• source/src/http_server/api/v1/health.c
• source/src/http_server/api/v2/metrics.c
• source/lib/monkey/mk_server/mk_lib.c
• README.md

The FluentDo Agent incorporates an HTTP server based on the Monkey HTTP Server to expose internal metrics and health status. This API is crucial for monitoring the agent's performance, data flow, and overall health in a production environment. It provides endpoints for metrics in both JSON and Prometheus formats, as well as a simple health check endpoint.

The agent is a downstream distribution of Fluent Bit, and its monitoring capabilities are essential for observability. The HTTP server runs in its own set of worker threads and communicates with the main agent engine asynchronously via message queues, ensuring that monitoring requests do not block the core logging pipeline.

Sources: README.md, source/lib/monkey/mk_server/mk_lib.c:45-56

Architecture

The monitoring system is built upon the Monkey HTTP Server, which is integrated as a library. The architecture relies on an asynchronous message passing system to decouple the data-producing components (e.g., input/output plugins) from the data-consuming HTTP API endpoints.

This diagram illustrates the high-level data flow for monitoring information.

Key components of this architecture include:

• Monkey HTTP Server (mk_server): The underlying web server that handles HTTP connections and requests.
• Message Queues (mk_mq_t): Used to send metrics data from the agent's core components to the HTTP server worker threads. Separate queues exist for v1 metrics, v2 metrics, and health checks.
• HTTP Handlers: Callback functions (e.g., cb_metrics_prometheus, cb_health) registered to specific URL paths (endpoints).
• In-Memory Buffering: Each HTTP worker thread maintains a list of the most recent metrics data received from the message queue. This buffer is used to serve API requests. A reference counting system and cleanup mechanism prevent memory leaks.

Sources: source/lib/monkey/mk_server/mk_lib.c:510-515, source/src/http_server/api/v1/metrics.c:150-174, source/src/http_server/api/v1/health.c:215-229, source/src/http_server/api/v2/metrics.c:140-154

Metrics API

The Metrics API exposes detailed performance counters from input and output plugins. It is available in two versions, v1 and v2, with support for both JSON and Prometheus formats.

V1 Metrics API

The v1 API provides metrics based on an internal msgpack format.

Sources: source/src/http_server/api/v1/metrics.c

Data Flow

The following diagram shows the sequence of events for handling v1 metrics.

Endpoints

Endpoint	Description	Output Format
/api/v1/metrics	Provides a snapshot of all plugin metrics.	JSON
/api/v1/metrics/prometheus	Provides metrics formatted for Prometheus consumption.	Prometheus Text

Sources: source/src/http_server/api/v1/metrics.c:640-643

Prometheus Formatting (/api/v1/metrics/prometheus)

The cb_metrics_prometheus function is responsible for transforming the internal msgpack metrics into the Prometheus text-based format.

The process involves:

1. Retrieving the latest raw msgpack metric buffer.
2. Iterating through the nested map structure of input and output metrics.
3. Constructing metric strings in the format fluentbit_<type>_<metric>_total{name="<plugin_name>"}.
4. Sorting all generated metric strings alphabetically.
5. Generating # HELP and # TYPE lines for each new metric family. The help text is derived from the metric name via the metrics_help_txt function.
6. Appending standard metrics like fluentbit_uptime, process_start_time_seconds, and fluentbit_build_info.
7. Sending the complete text payload as the HTTP response.

Sources: source/src/http_server/api/v1/metrics.c:303-599

This flowchart shows the logic for generating the Prometheus response.

V2 Metrics API

The v2 API is a more modern implementation that uses the cmetrics library format internally. This provides a more structured and efficient way to handle metrics.

Sources: source/src/http_server/api/v2/metrics.c

Data Flow

The v2 data flow is similar to v1 but uses cmetrics contexts instead of raw msgpack-to-JSON conversion.

Endpoints

Endpoint	Description	Output Format
/api/v2/metrics	Provides a snapshot of all plugin metrics.	Text (from cmt_encode_text_create)
/api/v2/metrics/prometheus	Provides metrics formatted for Prometheus consumption.	Prometheus Text (from cmt_encode_prometheus_create)

Sources: source/src/http_server/api/v2/metrics.c:244-249

Health Check API

The Health Check API provides a simple way to determine if the agent is operating correctly. It bases its status on the number of output plugin errors and retry failures over a configurable period.

Sources: source/src/http_server/api/v1/health.c

Endpoint

Endpoint	Description	Success Response	Failure Response
/api/v1/health	Returns the health status of the agent.	200 OK with body ok\n	500 Internal Server Error with body error\n

Sources: source/src/http_server/api/v1/health.c:255-269

Health Evaluation Logic

The health status is determined by the is_healthy() function. It works as follows:

1. A dedicated message queue, qid_health, receives metrics updates every second.
2. The cb_mq_health callback processes these messages, extracting errors and retries_failed from output plugins.
3. It maintains a list of these counters for a duration defined by the health_check_period configuration. Old entries are purged by cleanup_metrics.
4. When the /api/v1/health endpoint is called, is_healthy() calculates the number of errors and retry failures that occurred during the current period.
5. It compares these numbers against the configured limits (hc_errors_count and hc_retry_failure_count).
6. If the counts are within the limits, the agent is considered healthy (returns FLB_TRUE). Otherwise, it is unhealthy (returns FLB_FALSE).

The following state machine illustrates the health check logic.

Buffering and Memory Management

To efficiently serve metrics without querying plugins on every request, the HTTP server workers cache the latest metrics data.

• flb_hs_buf / flb_hs_hc_buf: A struct used to store a snapshot of metrics data.
• Buffer List: Each worker thread maintains a mk_list of these buffers.
• users counter: A reference counter in the buffer struct. It's incremented when an API callback starts using a buffer and decremented when it's done.
• cleanup_metrics(): This function is called periodically to garbage collect old buffers. A buffer is removed if it's not the latest one in the list and its users count is 0.

This mechanism ensures that API requests can be served quickly from memory while also preventing unbounded memory growth from accumulating old metric snapshots.

Field in flb_hs_buf	Type	Description
_head	struct mk_list	List head for linking buffers.
users	int	A reference counter to prevent deletion while in use.
data	flb_sds_t	The JSON formatted string data (v1).
raw_data	void *	The raw msgpack data (v1) or cmt context (v2).
raw_size	size_t	The size of the raw_data.

Sources: source/src/http_server/api/v1/metrics.c:70-96, 110-148, source/src/http_server/api/v1/health.c:67-85