Advanced Feature: Log Sampling

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/processor_sampling/sampling.c
• source/plugins/processor_log_sampling/log_sampling.c
• README.md
• source/README.md
• source/lib/monkey/CMakeLists.txt

Introduction

FluentDo Agent includes a native log sampling capability, designed as an advanced feature to help manage and reduce log volume in high-volume environments. This is crucial for controlling costs associated with data storage and processing, without losing complete visibility. The feature is implemented through processor plugins that can be configured to discard a certain percentage of log records, either randomly or deterministically based on record content.

This functionality is provided by two distinct processor plugins: sampling and log_sampling. Each offers a different strategy for reducing log data. The sampling processor provides fine-grained control based on a specific attribute within a log record, making it ideal for session-based or entity-based sampling. The log_sampling processor offers a simpler, percentage-based sampling mechanism that can be applied consistently to records sharing a common key.

Sources: README.md, source/README.md, source/plugins/processor_sampling/sampling.c, source/plugins/processor_log_sampling/log_sampling.c

Architecture

Log sampling is implemented within the agent's data pipeline as a "Processor" plugin. Processors intercept data chunks, apply transformations or filtering, and then pass the data to the next stage. The sampling processors decide whether to keep or drop each individual record within a data chunk.

The overall data flow with a sampling processor is as follows:

This diagram illustrates how a sampling processor fits into the FluentDo Agent pipeline, intercepting data chunks and selectively forwarding records.

Sources: source/plugins/processor_sampling/sampling.c:135-156, source/plugins/processor_log_sampling/log_sampling.c:104-121

FluentDo Agent provides two different plugins for this purpose: sampling and log_sampling.

The sampling Processor

The sampling processor is designed for attribute-based sampling. It allows you to define a sample rate that is deterministically applied based on the value of a specific field (attribute) in the record. This is useful for scenarios where you want to keep all logs for a sampled subset of entities, such as user IDs or trace IDs.

Sources: source/plugins/processor_sampling/sampling.c:27-28

Configuration

The sampling processor is configured in the [PROCESSOR] section of the FluentDo Agent configuration file.

Parameter	Type	Default	Description
Name	String	-	Must be sampling.
Rate	Integer	100	The percentage (0-100) of records to keep if the specified Attribute is found.
Attribute	Key	NULL	The record attribute (key) whose value will be used for the sampling decision.
Default_Rate	Integer	0	The percentage (0-100) of records to keep if the Attribute is not found in the record. This sampling is random.

Sources: source/plugins/processor_sampling/sampling.c:33-47

Logic Flow

The processor's logic changes based on whether the Attribute parameter is configured.

1. Attribute-Based (Deterministic) Sampling: If Attribute is set, the processor hashes the attribute's value. If the hash modulo 100 is less than Rate, the record is kept. This ensures that all records with the same attribute value are either all kept or all dropped. If the attribute is not found in a record, the Default_Rate is used with a random decision.
2. Random Sampling: If Attribute is not set, the processor falls back to simple random sampling, keeping records based on the Rate percentage.

The following diagram illustrates the decision logic of the sampling processor.

The log_sampling Processor

The log_sampling processor provides a simpler, deterministic sampling method. It hashes either a specified key's value or the entire record body to decide whether to keep or drop a record based on a single sample rate. This is useful for general log reduction while maintaining consistency for related log entries.

Sources: source/plugins/processor_log_sampling/log_sampling.c:27-28

Configuration

The log_sampling processor is configured in the [PROCESSOR] section of the FluentDo Agent configuration file.

Parameter	Type	Default	Description
Name	String	-	Must be log_sampling.
Sample_Rate	Integer	10	The percentage (0-100) of records to keep.
Key	Key	NULL	If set, the value of this key is hashed for the sampling decision. If not set, or if the key is not found, the entire record body is hashed.

Sources: source/plugins/processor_log_sampling/log_sampling.c:33-43

Logic Flow

The processor's logic is consistently deterministic, based on the Sample_Rate.

1. The processor determines what content to hash:
   • If the Key property is configured and present in the record, its value is used.
   • Otherwise, the entire raw record body is used.
2. The selected content is hashed.
3. If the hash modulo 100 is less than Sample_Rate, the record is kept. Otherwise, it is dropped.

The following diagram shows the decision logic for the log_sampling processor.

Build-Time Configuration

In FluentDo Agent, plugins can often be enabled or disabled at build time through CMake options. Based on the project's structure, it is likely that the sampling plugins can be included or excluded from a build using flags. This allows for creating optimized agent binaries that only include necessary features.

A typical pattern for this in the build system is:

option(MK_PLUGIN_LOGGER "Log Writer" No)

It can be inferred that the sampling plugins would follow a similar convention, for example:

option(MK_PLUGIN_SAMPLING "Enable sampling processor" Yes)
option(MK_PLUGIN_LOG_SAMPLING "Enable log_sampling processor" Yes)

Sources: source/lib/monkey/CMakeLists.txt:37-53

Conclusion

The log sampling features in FluentDo Agent offer powerful tools for managing data volume and controlling costs. By providing two distinct plugins—sampling for attribute-driven deterministic sampling and log_sampling for simpler percentage-based deterministic sampling—the agent gives users flexible options to tailor their data reduction strategy to specific needs. Proper configuration of these processors can lead to significant reductions in log traffic while preserving the integrity of sampled sessions or ensuring a consistent representation of the overall data.

Sources: README.md, source/plugins/processor_sampling/sampling.c, source/plugins/processor_log_sampling/log_sampling.c