AWS S3 Integration
Relevant source files
The following files were used as context for generating this wiki page:
AWS S3 Integration provides infrahub-backup with the capability to store and retrieve backup archives in S3-compatible object storage. The system supports standard AWS S3, MinIO, and Google Cloud Storage via an S3-compatible API. This integration allows for automated periodic backups in Kubernetes environments, ensuring disaster recovery artifacts persist independently of the cluster.
Sources: specs/001-k8s-helm-backup/plan.md, tools/s3-uploader/main.go:23-28
Architecture and Components
The S3 integration uses the minio-go/v7 library to handle object storage operations. This library is selected over the standard AWS SDK to ensure compatibility with Google Cloud Storage and other S3-compliant providers that may have specific header requirements.
Key Data Structures
The system organizes S3 settings and operations through two primary structures:
- S3Config: Stores bucket names, key prefixes, custom endpoints, and regional information.
- S3Client: Wraps the underlying MinIO client and provides high-level methods for Infrahub backup operations.
Sources: tools/s3-uploader/main.go:16-21, tools/s3-uploader/main.go:29-32
S3 Client Lifecycle
The following diagram illustrates the initialization and execution flow of the S3 client:
The diagram shows how the NewS3Client function configures routing styles and credentials before performing storage operations.
Sources: tools/s3-uploader/main.go:35-81, specs/004-backup-retention-policy/tasks.md:T007-T008
Core Operations
Backup Upload
When the create command is issued with the --s3-upload flag, the system performs a backup and then initiates an upload. The S3 key is constructed by joining the configured prefix with the local filename. The client uses multipart uploads automatically for large files.
Sources: tools/s3-uploader/main.go:94-118, specs/001-k8s-helm-backup/plan.md:2.3
Restore and Download
The restore process supports s3://bucket/key URI formats. To prevent data corruption or accidental deletion of local files during restore, the system downloads S3 artifacts to a collision-proof temporary path in the BackupDir (e.g., restore-latest-*.download) before performing the restore.
Sources: specs/005-restore-latest-backup/tasks.md:T009, specs/001-k8s-helm-backup/plan.md:2.4
Retention and Pruning
The S3 integration supports retention policies by listing objects with a specific prefix and identifying prunable candidates based on their timestamps. The S3Client includes List and Delete methods to facilitate automated cleanup of old backups.
Sources: specs/004-backup-retention-policy/tasks.md:T007-T008
Configuration Options
S3 integration is configured via environment variables, CLI flags, or Kubernetes Secrets in Helm deployments.
| Parameter | CLI Flag | Environment Variable | Description |
|---|---|---|---|
| Bucket | --s3-bucket | INFRAHUB_S3_BUCKET | The name of the S3 bucket. |
| Prefix | --s3-prefix | INFRAHUB_S3_PREFIX | Optional folder/prefix for S3 keys. |
| Endpoint | --s3-endpoint | INFRAHUB_S3_ENDPOINT | Custom URL for MinIO/GCS. |
| Region | --s3-region | INFRAHUB_S3_REGION | AWS region (default: us-east-1). |
| Access Key | N/A | AWS_ACCESS_KEY_ID | S3 access credential. |
| Secret Key | N/A | AWS_SECRET_ACCESS_KEY | S3 secret credential. |
Sources: specs/001-k8s-helm-backup/plan.md:2.1, tools/s3-uploader/main.go:127-133
Technical Implementation Details
The S3 client implements a credentials chain that attempts to load authentication data from three sources in order:
- Environment variables (
EnvAWS). - Local AWS credential files (
FileAWSCredentials). - IAM instance roles (
IAM).
Sources: tools/s3-uploader/main.go:63-67
// src/internal/app/s3.go or tools/s3-uploader/main.go:63-67
creds := credentials.NewChainCredentials([]credentials.Provider{
&credentials.EnvAWS{},
&credentials.FileAWSCredentials{},
&credentials.IAM{},
})
Sources: tools/s3-uploader/main.go:63-67
For S3-compatible providers like MinIO, the client automatically enables path-style bucket lookups when a custom endpoint is provided. This ensures compatibility with services that do not support virtual-host style routing.
Sources: tools/s3-uploader/main.go:48-59
AWS S3 Integration centralizes remote storage logic to provide Infrahub with durable, cross-environment backup capabilities. By abstracting storage behind a specialized client and supporting standard S3 protocols, the system enables reliable disaster recovery for both Docker and Kubernetes deployments.