Advanced Feature: Log Deduplication
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Log Deduplication is a native, built-in feature of the FluentDo Agent designed to reduce log volume by identifying and consolidating duplicate log entries. As a hardened downstream distribution of Fluent Bit, FluentDo Agent includes this advanced capability to help users manage high-volume environments, optimize storage, and reduce data transfer costs significantly—by up to 40%. This feature operates as a processor plugin within the agent's data pipeline, making it a seamless and performant part of the log processing workflow.
Sources: README.md
Architectural Role and Data Flow
The Log Deduplication feature is implemented as a custom processor plugin within the FluentDo Agent. This means it is not part of the standard upstream Fluent Bit but is a specific enhancement provided by FluentDo. Its position as a processor allows it to intercept the log stream after parsing and before filtering or buffering, where it can efficiently identify and aggregate duplicate records.
Sources: README.md, scripts/create-sync-pr.sh, scripts/sync-upstream-patches.sh
The following diagram illustrates the data flow through the agent, highlighting the position of the deduplication processor.
This diagram shows the typical data pipeline in FluentDo Agent, with the Log Deduplication processor acting on the data stream after parsing.
Key Capabilities
The primary goal of this feature is to reduce data volume without losing information about the frequency of events.
| Capability | Description |
|---|---|
| Native Implementation | Built directly into the agent in C for maximum performance and a small footprint, consistent with the project's goals. |
| Log Volume Reduction | Identifies and combines identical log records, replacing them with a single record and a count of occurrences. |
| Cost Savings | Can reduce logging-related costs (storage, transfer, indexing) by up to 40% in environments with repetitive logs. |
| Seamless Integration | Operates as a standard processor plugin, making it easy to enable and configure within the existing pipeline. |
Sources: README.md
Configuration
The specific implementation files for the deduplication processor (dedup.c, dedup.h) were not included in the source analysis for this document. Therefore, a detailed list of configuration parameters cannot be provided.
However, based on the standard Fluent Bit plugin architecture, configuration would typically be managed within a [PROCESSOR] block in the FluentDo Agent configuration file. It would likely include parameters to define which fields to use for hashing, the size of the tracking cache, and the time window for deduplication.
Development and Quality Standards
While the direct source code for the deduplication feature is not analyzed here, the project's repository contains contribution guidelines for its dependencies, which shed light on the high standards applied to custom features.
- Code Style and Portability: The project emphasizes a consistent C coding style (based on C90/C99), enforced with tools like
clang-format. This ensures the deduplication feature is maintainable and portable. - Performance: Performance is a critical consideration. The contribution guidelines for included libraries highlight the importance of extensive benchmarking and avoiding performance regressions. This focus is essential for a real-time processor like the deduplication plugin.
- Static Analysis and Testing: The use of static analysis tools (
scan-build) and a "zero-warning tolerance" policy suggests that custom features are subject to rigorous correctness checks to prevent bugs. - API/ABI Stability: The project maintains strict API and ABI compatibility guarantees, ensuring that features like the deduplication processor will not introduce breaking changes for users.
Sources: source/lib/librdkafka-2.10.1/CONTRIBUTING.md, source/lib/zstd-1.5.7/CONTRIBUTING.md
Upstream Synchronization and Maintenance
The Log Deduplication feature is a custom enhancement not present in upstream Fluent Bit. The project's maintenance scripts show a clear process for syncing with upstream while ensuring custom features remain functional.
The sync-upstream-patches.sh script categorizes upstream commits to selectively apply relevant fixes, implicitly skipping changes that would conflict with custom features. The create-sync-pr.sh script generates a pull request template that includes a verification checklist for FluentDo customizations, confirming that the deduplication feature must be explicitly tested and verified during every upstream sync.
Sources: scripts/sync-upstream-patches.sh, scripts/create-sync-pr.sh
The following diagram shows the workflow for syncing with upstream and verifying custom features like deduplication.
This diagram illustrates the development workflow for keeping FluentDo Agent in sync with upstream Fluent Bit while ensuring custom features are not broken.
Sources: scripts/create-sync-pr.sh
Conclusion
The Log Deduplication feature is a powerful, performance-oriented enhancement in FluentDo Agent. By operating as a native processor plugin, it provides a significant reduction in log data volume and associated costs. Its status as a custom feature is carefully managed through a disciplined development and upstream synchronization process, ensuring it remains a stable and reliable component of this hardened Fluent Bit distribution.