Advanced Feature: AI-Powered Routing
Relevant source files
The following files were used as context for generating this wiki page:
AI-Powered Routing is an advanced, built-in feature of the FluentDo Agent. It is designed to provide intelligent, content-aware routing of log data, which is described as "Smart log routing". This capability is one of the key differentiators that extends the functionality of the core Fluent Bit engine upon which the agent is built.
The primary goal of this feature is to enable more sophisticated data pipelines where logs can be directed to different destinations based on their content, as determined by AI processors. This allows for cost optimization, improved security, and more efficient log analysis by sending only relevant data to specific systems.
Sources: README.md:15, source/README.md:15, README.md:60, source/README.md:60
Feature Overview
AI-Powered Routing is listed as one of several advanced, production-ready capabilities native to the FluentDo Agent. These features are designed to enhance the agent's performance and utility in enterprise environments.
Sources: README.md:55-62, source/README.md:55-62
Key Advanced Features
The table below lists the advanced features mentioned in the project documentation, placing AI-based routing in the context of other native enhancements.
| Feature | Description |
|---|---|
| Native de-duplication | Reduces log volume by eliminating duplicate entries. |
| Native log sampling | Reduces log volume by sampling high-volume data streams. |
| AI processors for routing | Provides "Smart log routing" and "AI-based filtering". |
| Enhanced storage layer | Offers more efficient buffering of data. |
| Native flattening | For OpenSearch/Elasticsearch destinations. |
| Type safety | Includes automatic conflict resolution. |
Sources: README.md:13-16, source/README.md:13-16, README.md:58-62, source/README.md:58-62
Architecture and Implementation
The provided source files do not contain the specific implementation details, source code, or configuration parameters for the AI-Powered Routing feature. The information available is limited to high-level descriptions in the project's README files.
However, the project's build system provides insight into how such a feature would be integrated. The FluentDo Agent, building on the Monkey web server framework, utilizes a plugin-based architecture. Advanced features like AI routing are likely implemented as processor plugins that can be enabled or disabled at build time.
Sources: source/lib/monkey/CMakeLists.txt:43-60
General Plugin Architecture
The following diagram illustrates the general plugin architecture of the underlying Monkey framework, which is likely leveraged for custom processors in FluentDo Agent.
This diagram shows a conceptual flow where logs enter the agent, are passed to a processing pipeline, and an AI processor plugin makes a routing decision to direct the data to one of multiple outputs.
Sources: source/lib/monkey/CMakeLists.txt:43-60
Build-Time Configuration
The CMakeLists.txt for the Monkey dependency shows a series of option directives used to enable or disable plugins at compile time. A similar mechanism is likely used for the FluentDo Agent's proprietary features, including the AI processor.
| CMake Option | Description | Default |
|---|---|---|
MK_PLUGIN_AUTH | Basic authentication | No |
MK_PLUGIN_CGI | CGI support | No |
MK_PLUGIN_LOGGER | Log Writer | No |
MK_PLUGIN_TLS | TLS/SSL support | No |
MK_PLUGIN_MANDRIL | Security | Yes |
Sources: source/lib/monkey/CMakeLists.txt:43-57
Development and Contribution
The development process for new features and patches, including potential updates to the AI routing feature, follows a structured workflow. Contributions are managed via pull requests against the dev branch and must pass a suite of CI tests and static analysis.
Sources: source/lib/zstd-1.5.7/CONTRIBUTING.md:10-16, source/lib/librdkafka-2.10.1/CONTRIBUTING.md:103-110
Sync and Release Process
Upstream changes from Fluent Bit are periodically synchronized into the FluentDo Agent. A script, create-sync-pr.sh, is used to automate the creation of pull requests for these syncs. This process ensures that the agent stays current with the upstream project while maintaining its custom features like AI-powered routing.
The following diagram illustrates the sync process.
This diagram shows the developer-driven process for syncing upstream changes and creating a pull request, which is managed by a dedicated script.
Sources: scripts/create-sync-pr.sh
Conclusion
AI-Powered Routing is a key advanced feature of the FluentDo Agent, advertised to provide "Smart log routing". While the provided source files confirm its existence as a high-level feature, they do not contain the specific source code, configuration options, or detailed documentation for its implementation. The agent's plugin-based architecture, inherited from its underlying components, is the likely mechanism for integrating this functionality. For detailed information, the source code for the specific AI processor plugin would need to be analyzed.
Sources: README.md:15,60, source/README.md:15,60