Output Plugin: Grafana Loki
Relevant source files
The following files were used as context for generating this wiki page:
The Grafana Loki output plugin, out_loki, is an integrated component of the FluentDo Agent designed to send log records to a Grafana Loki service. As a standard output plugin, it is responsible for formatting the data received from the agent's core engine and transmitting it to the configured Loki endpoint.
The plugin is registered within the agent's build system, ensuring it is available as a destination for log data pipelines. It leverages the agent's core functionalities for data transport and serialization, likely using included libraries for creating JSON payloads and handling HTTP communication.
Sources: source/plugins/CMakeLists.txt:232, README.md
Plugin Registration
The out_loki plugin is registered into the FluentDo Agent core during the build process. The CMakeLists.txt file for plugins contains a macro call that makes the plugin available to the engine.
# source/plugins/CMakeLists.txt:232
REGISTER_OUT_PLUGIN("out_loki")
This registration is handled by the REGISTER_OUT_PLUGIN macro, which adds the plugin's definition, loki_plugin, to the list of available output plugins in the configuration.
The following diagram illustrates the plugin registration process.
Configuration
The out_loki plugin is configured like any other output plugin in FluentDo Agent, either through command-line arguments or a configuration file. The core engine parses these options and applies them to the plugin instance.
Sources: source/src/fluent-bit.c:648-847
Command-Line Options
The following command-line options are used to enable and configure the out_loki plugin.
| Flag | Long Option | Description | Example |
|---|---|---|---|
-o | --output | Sets an output plugin. | -o loki |
-m | --match | Sets a match pattern to select records for this plugin. | -m 'my_app.*' |
-p | --prop | Sets a configuration property for the plugin. | -p 'url=http://localhost:3100' |
-t | --tag | Sets a tag for the plugin, equivalent to -p tag=abc. | -t loki.service |
Sources: source/src/fluent-bit.c:134-162, 770-811
When using these flags, the agent creates an [OUTPUT] section in its internal configuration context. For example, the command fluent-bit -i cpu -o loki -p url=http://loki:3100 would result in an internal configuration for the Loki plugin.
Architecture and Data Flow
The out_loki plugin functions as the final stage in the logging pipeline before data is sent to an external Loki service. It receives data chunks from the engine, formats them, and sends them over the network.
This diagram shows the conceptual data flow from a log source to Grafana Loki.
Sources: source/src/fluent-bit.c:1030-1065, README.md
- Data Ingestion: Input plugins collect data.
- Processing: The core engine routes data through any configured filter plugins.
- Routing: The engine matches records against the
out_lokiplugin'sMatchrule. - Formatting: The
out_lokiplugin receives the records and is responsible for transforming them into the JSON format required by the Loki push API. - Transport: The plugin uses an HTTP client to send the formatted JSON payload to the configured Loki server URL.
Payload Formatting with yyjson
To send data to Loki, the plugin must construct a JSON payload. The FluentDo Agent project includes the yyjson library, a high-performance C JSON library, which is likely used for this purpose. yyjson is designed for speed and portability, complying with the C89 standard.
Key features of yyjson relevant to the Loki plugin include:
- Fast Serialization: Capable of writing gigabytes of JSON data per second.
- Mutable Documents: Allows for the creation and modification of JSON objects and arrays in memory before serialization.
- Developer-Friendly API: Provides functions to easily add strings, integers, and nested objects/arrays.
Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/index.html
Example: Creating a JSON Payload
The following example, derived from the yyjson documentation, demonstrates how a JSON object can be constructed. The out_loki plugin would use similar logic to build the payload for Loki.
// Create a mutable doc
yyjson_mut_doc *doc = yyjson_mut_doc_new(NULL);
yyjson_mut_val *root = yyjson_mut_obj(doc);
yyjson_mut_doc_set_root(doc, root);
// Create a "streams" array
yyjson_mut_val *streams = yyjson_mut_arr(doc);
yyjson_mut_obj_add_val(doc, root, "streams", streams);
// Create a single stream object
yyjson_mut_val *stream_obj = yyjson_mut_obj(doc);
yyjson_mut_arr_append(streams, stream_obj);
// Add labels to the stream
yyjson_mut_val *stream_labels = yyjson_mut_obj(doc);
yyjson_mut_obj_add_str(doc, stream_labels, "job", "fluentdo");
yyjson_mut_obj_add_val(doc, stream_obj, "stream", stream_labels);
// Add values (log lines) to the stream
yyjson_mut_val *values = yyjson_mut_arr(doc);
yyjson_mut_obj_add_val(doc, stream_obj, "values", values);
// Add a log entry: [timestamp, line]
yyjson_mut_val *log_entry = yyjson_mut_arr(doc);
yyjson_mut_arr_add_str(doc, log_entry, "1640995200000000000");
yyjson_mut_arr_add_str(doc, log_entry, "level=info message=\"log line content\"");
yyjson_mut_arr_append(values, log_entry);
// Serialize to string
const char *json = yyjson_mut_write(doc, 0, NULL);
if (json) {
printf("json: %s\n", json);
free((void *)json);
}
// Free the doc
yyjson_mut_doc_free(doc);
Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/index.html
Data Transport
The formatted JSON payload is sent to the Loki server via an HTTP POST request. The agent's codebase includes dependencies like nghttp2, indicating support for modern and efficient HTTP/2 communication, which would be beneficial for high-throughput logging environments. The underlying HTTP client handles the network connection, request serialization, and response handling.