Filter Plugin: Kubernetes
Relevant source files
The following files were used as context for generating this wiki page:
The Kubernetes filter plugin is a crucial component for log processing in containerized environments. Its primary function is to enrich log records with valuable Kubernetes metadata. When logs are collected from pods, this filter intercepts them and appends information such as the pod's name, namespace, container ID, labels, and annotations. This enrichment provides essential context, making logs significantly more useful for debugging, monitoring, and analysis.
The plugin works by parsing the log's tag to identify the source pod. It then queries the Kubernetes API Server or the local Kubelet to fetch the relevant metadata. To optimize performance and reduce the load on the Kubernetes control plane, the fetched metadata is cached in memory for a configurable duration (TTL). The plugin also supports advanced features like dynamically applying parsers based on pod annotations and merging nested JSON log strings into the main log record.
Configuration
The behavior of the Kubernetes filter is controlled by a set of configuration properties defined in the [FILTER] section of the Fluent Bit configuration file. The creation and initialization of these settings are handled in flb_kube_conf_create.
Sources: source/plugins/filter_kubernetes/kube_conf.c, source/plugins/filter_kubernetes/kube_conf.h
Key Configuration Options
| Property | Description | Default Value |
|---|---|---|
Kube_URL | URL of the Kubernetes API server. | https://kubernetes.default.svc:443 |
Kube_Tag_Prefix | The prefix in the log tag that identifies it as a Kubernetes log. | kube.var.log.containers. |
Use_Kubelet | If enabled, the plugin will get metadata from the local Kubelet instead of the API server. | Off |
Kubelet_Port | TCP port for the Kubelet API. | 10250 |
Merge_Log | If a log entry is a JSON string, merge its keys and values into the main record. | Off |
Merge_Log_Key | If Merge_Log is enabled, this specifies the key containing the JSON string. | log |
Keep_Log | If Merge_Log is enabled, keep the original Merge_Log_Key field after merging. | On |
Labels | Include Kubernetes labels in the metadata. | On |
Annotations | Include Kubernetes annotations in the metadata. | `On`` |
K8S-Logging.Parser | If enabled, allows pods to suggest a parser via the fluentbit.io/parser annotation. | Off |
K8S-Logging.Exclude | If enabled, allows pods to be excluded from logging via the fluentbit.io/exclude annotation. | Off |
Dummy_Meta | For testing, injects dummy metadata if real metadata is not found. | Off |
Cache_Use_Docker_Id | Use the full Docker container ID for the cache key. | Off |
Kube_Meta_Cache_TTL | TTL in seconds for cached pod metadata. | 0 (no expiration) |
Kube_Token_File | Path to the Kubernetes service account token file. | /var/run/secrets/kubernetes.io/serviceaccount/token |
Kube_CA_File | Path to the Kubernetes CA certificate file. | /var/run/secrets/kubernetes.io/serviceaccount/ca.crt |
Sources: source/plugins/filter_kubernetes/kube_conf.c:116-368, source/plugins/filter_kubernetes/kube_conf.h:40-197
Core Architecture and Data Flow
The plugin integrates into the Fluent Bit pipeline through standard filter callbacks: cb_kube_init, cb_kube_filter, and cb_kube_exit.
cb_kube_init: Called upon pipeline startup. It creates the plugin's configuration context (struct flb_kube) by callingflb_kube_conf_create, which sets up connections to the Kubernetes API, initializes caches, and prepares regex parsers.cb_kube_exit: Called on shutdown. It cleans up resources by callingflb_kube_conf_destroy.cb_kube_filter: This is the main processing function, executed for each log record that matches the filter's rule.
Sources: source/plugins/filter_kubernetes/kubernetes.c:753-839
Filtering Logic
The diagram below illustrates the step-by-step process inside the cb_kube_filter function.
This diagram shows the data flow for enriching a log record with Kubernetes metadata.
Sources: source/plugins/filter_kubernetes/kubernetes.c:510-750
Metadata Fetching and Caching
To avoid overwhelming the Kubernetes API server, the plugin implements a robust caching layer for pod and namespace metadata.
Sources: source/plugins/filter_kubernetes/kube_meta.c, source/plugins/filter_kubernetes/kube_conf.h
Metadata Sources
The plugin can fetch metadata from two primary sources, controlled by the Use_Kubelet configuration:
- Kubernetes API Server: The default source. The plugin communicates with the API server over HTTPS to fetch pod and namespace details. This requires proper service account permissions (RBAC).
- Kubelet: An alternative source. If
Use_Kubeletis enabled, the plugin queries the/podsendpoint on the local node's Kubelet. This is often faster and reduces load on the central API server, but it can only provide information about pods running on the same node.
Caching Mechanism
Metadata is stored in two in-memory hash tables within the flb_kube context:
hash_table: Caches pod-level metadata, keyed by a string derived from the log tag (e.g.,namespace_pod-name_container-name).namespace_hash_table: Caches namespace-level metadata (labels and annotations), keyed by the namespace name.
The kube_meta_cache_ttl option defines how long cached entries remain valid. A TTL of 0 means entries never expire.
The sequence diagram below shows the interaction for a cache miss.
This diagram illustrates the process of fetching and caching metadata when it is not found in the local cache.
Sources: source/plugins/filter_kubernetes/kube_meta.c:36-100, source/plugins/filter_kubernetes/kube_conf.h:182-184
Key Data Structures
The plugin relies on several key C structures to manage its state and data.
This diagram shows the main context structure flb_kube and the kube_meta structure it uses to store cached metadata.
Sources: source/plugins/filter_kubernetes/kube_conf.h:92-197, [source/plugins/filter_kubernetes/kube_meta.c:25-34]
Special Features
Pod-level Configuration via Annotations
The plugin can be dynamically configured on a per-pod basis using Kubernetes annotations. This allows developers to control logging behavior without modifying the central Fluent Bit configuration.
fluentbit.io/parser: <parser_name>: If theK8S-Logging.Parseroption is enabled, this annotation instructs the plugin to apply a specific pre-defined parser to the logs from that pod.fluentbit.io/exclude: "true": If theK8S-Logging.Excludeoption is enabled, this annotation causes the plugin to drop all logs from that pod.
Sources: source/plugins/filter_kubernetes/kube_conf.h:103-104, source/tests/runtime/filter_kubernetes.md
Merge Log
When containerized applications output logs as JSON strings, they often get encapsulated within another JSON structure by the container runtime (e.g., Docker). The Merge_Log feature addresses this by parsing the inner JSON string and promoting its fields to the top level of the log record.
Sources: source/plugins/filter_kubernetes/kube_conf.h:112-125
AWS EKS Integration
The plugin contains specific logic for integration with Amazon EKS, including the ability to map pods to their corresponding Kubernetes services. This is managed through configuration options like aws_use_pod_association and platform. When platform is detected or set to eks, the plugin can query an association endpoint to enrich logs with service-level metadata.
Sources: source/plugins/filter_kubernetes/kube_conf.h:160-193
Plugin Registration
The Kubernetes filter is registered within the Fluent Bit build system using the REGISTER_FILTER_PLUGIN macro in the plugins/CMakeLists.txt file. This ensures it is compiled and linked into the final fluent-bit binary.
# FILTERS
# =======
...
REGISTER_FILTER_PLUGIN("filter_kubernetes")
...
Sources: source/plugins/CMakeLists.txt:402
Conclusion
The Kubernetes filter plugin is an indispensable tool for operating Fluent Bit in a Kubernetes environment. By automatically enriching logs with critical metadata, it transforms raw container output into structured, queryable, and context-rich data. Its caching mechanisms ensure high performance, while features like annotation-based configuration provide flexibility for developers. This combination of power and efficiency makes it a cornerstone of any Kubernetes logging strategy.