Output Plugin: Amazon S3

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/out_s3/s3.c
• source/plugins/out_s3/s3.h
• source/plugins/out_s3/s3_store.h
• source/src/fluent-bit.c
• source/README.md

The Amazon S3 output plugin allows shipping log data to an Amazon S3 bucket. It is a core component of the FluentDo Agent, which is a hardened distribution of Fluent Bit. The plugin is designed for high performance and resilience, featuring local file system buffering, flexible object key formatting, and support for both single-part and multipart uploads to handle various data volumes efficiently.

This plugin can handle both standard log events and larger binary data blobs, making it a versatile solution for data archival and integration with data lakes built on S3. It includes robust authentication mechanisms, leveraging the standard AWS credential provider chain and supporting IAM role assumption via STS for secure cross-account access.

Architecture and Core Components

The plugin's architecture is centered around a local storage buffer, an upload manager that decides between different S3 APIs, and a set of timers to handle asynchronous operations and timeouts.

Key Data Structures

The plugin's state and operations are managed through several key C structures.

This diagram shows the main data structures used by the S3 output plugin and their relationships.

• struct flb_s3: The main context for the plugin instance. It holds configuration, AWS client details, local storage context, and lists for managing active uploads.
• struct multipart_upload: Manages the state of a single S3 multipart upload, including the upload_id, the S3 object key, and the ETags of completed parts.
• struct s3_file: Represents a single chunk of data buffered on the local filesystem before being uploaded.
• struct upload_queue: An entry in the queue used when preserve_data_ordering is enabled, ensuring chunks are uploaded sequentially.

Sources: source/plugins/out_s3/s3.h

Local Buffering

The plugin uses a local directory specified by store_dir to buffer data before sending it to S3. This provides resilience against network failures or backpressure.

1. When cb_s3_flush is called, the incoming data chunk is written to a file in the store_dir.
2. The s3_store_buffer_put() function handles the creation and writing to these local files.
3. Files are managed per tag, and the plugin attempts to aggregate data up to upload_chunk_size before initiating an upload.
4. The s3_store_init() function initializes this storage layer at startup, and s3_store_has_data() checks for leftover buffers from previous runs.

Sources: source/plugins/out_s3/s3.c:3032-3044, 3968-4024

Data Processing and Upload Flow

The core logic resides in the cb_s3_flush callback, which is invoked by the Fluent Bit engine when data is ready to be sent.

This diagram illustrates the data flow from receiving a chunk to uploading it to S3.

Sources: source/plugins/out_s3/s3.c:4027-4180

Upload Strategies

The plugin dynamically chooses between two S3 upload APIs based on configuration and data size.

1. PutObject API

This is a single-request upload used for smaller objects. The plugin opts for PutObject if:

• use_put_object is explicitly set to true.
• The data to be uploaded is smaller than the minimum multipart chunk size (5MB).
• An upload operation times out before enough data for a multipart upload is aggregated.

The s3_put_object function constructs and sends the PUT request to S3.

Sources: source/plugins/out_s3/s3.c:2038-2045, 2225-2336

2. Multipart Upload API

For larger files, the plugin uses the S3 Multipart Upload API, which is more resilient and efficient for streaming large amounts of data. This is the default behavior unless use_put_object is enabled.

The process is as follows:

This sequence diagram shows the interaction with the S3 API during a multipart upload.

• Initiation: create_multipart_upload() is called to send the initial request and get an UploadId.
• Part Upload: upload_part() sends individual data chunks (upload_chunk_size).
• Completion/Abortion: complete_multipart_upload() or abort_multipart_upload() finalizes the transaction.

This state is managed in the multipart_upload struct and persisted to the filesystem in the store_dir to survive restarts.

Sources: source/plugins/out_s3/s3.c:2603-2646, 2813-2856

Blob (File) Handling

The plugin can also process large files directly when an input plugin sends data in FLB_EVENT_TYPE_BLOBS format. This is useful for forwarding entire files to S3.

1. When a blob chunk is received, process_blob_chunk is called.
2. It uses a SQLite database (blob_database_file) to track the file and its parts.
3. The file is broken down into parts based on part_size.
4. A dedicated timer, cb_s3_upload_blob, processes these parts, uploading them using the multipart API.
5. Once all parts of a file are uploaded, the multipart upload is completed, and the entry is removed from the database.

This mechanism is designed for reliable, large-file transfer and is more sophisticated than the standard log buffering.

Sources: source/plugins/out_s3/s3.c:4182-4211, 3131-3529

Configuration Options

The plugin is configured through the [OUTPUT] section in the Fluent Bit configuration file.

Property	Type	Default	Description
bucket	string	-	S3 bucket name. Required.
region	string	us-east-1	AWS region for the S3 bucket.
total_file_size	size	100M	The final size of the object in S3. Minimum 1MB.
upload_chunk_size	size	5M	The size of each part for multipart uploads. Minimum 5MB.
upload_timeout	time	10m	If an upload is not completed within this time, it will be finalized regardless of size.
store_dir	string	/tmp/fluent-bit/s3	Directory for local data buffering.
store_dir_limit_size	size	0	The maximum size of the store_dir. 0 means unlimited.
s3_key_format	string	/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S	Format string for the S3 object key. Supports strftime, $TAG, $TAG[N], $INDEX, and $UUID.
s3_key_format_tag_delimiters	string	.	Characters used to split the tag for $TAG[N] expansion.
use_put_object	bool	false	If true, forces the use of the PutObject API instead of multipart uploads.
compression	string	null	Compression type. Supported: gzip, zstd, arrow, parquet.
content_type	string	null	Sets the Content-Type header for the S3 object.
canned_acl	string	null	A predefined Canned ACL policy for the S3 object (e.g., private, public-read).
storage_class	string	null	S3 storage class for the object (e.g., STANDARD, INTELLIGENT_TIERING).
role_arn	string	null	IAM role to assume for cross-account access.
endpoint	string	null	Custom endpoint for S3-compatible services.
sts_endpoint	string	null	Custom endpoint for the AWS STS service.
external_id	string	null	External ID to use when assuming an IAM role.
profile	string	null	AWS profile name from ~/.aws/credentials to use.
preserve_data_ordering	bool	true	Ensures chunks are uploaded in the order they are received to prevent data shuffling on retries.
log_key	string	null	If set, only the value of this key from the log record is sent to S3.
static_file_path	bool	false	If true, disables the automatic appending of a random string to the S3 key.
blob_database_file	string	null	Path to a SQLite database for tracking blob file uploads.
part_size	size	25M	Size of each part when uploading blob files.

Sources: source/plugins/out_s3/s3.c:4330-4569

Authentication

Authentication with AWS is handled by the flb_aws_client, which uses a provider chain to find credentials. The S3 plugin initializes this chain via flb_standard_chain_provider_create.

The default provider chain searches for credentials in this order:

1. Environment Variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).
2. Shared Credentials File (~/.aws/credentials).
3. IAM Role for the EC2 instance or ECS task.

The plugin also supports assuming an IAM role by configuring role_arn. This uses the AWS Security Token Service (STS) via flb_sts_provider_create to obtain temporary credentials.

Sources: source/plugins/out_s3/s3.c:1019-1094

Summary

The S3 output plugin provides a powerful and resilient method for exporting logs and files from FluentDo Agent to Amazon S3. Its support for local buffering ensures data is not lost during network interruptions. The dual-mode upload strategy, using either PutObject for simplicity or the multipart API for large-scale streaming, offers flexibility for different use cases. With comprehensive configuration for object key naming, authentication, and data formatting, it is a highly adaptable component for building data pipelines on AWS.