Object Storage with S3
Relevant source files
The following files were used as context for generating this wiki page:- packages/@aws-cdk/aws-s3/lib/bucket.ts
- packages/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts
- packages/@aws-cdk/aws-s3-notifications/lib/lambda.ts
- packages/@aws-cdk/aws-s3/README.md
- packages/@aws-cdk/aws-s3-deployment/README.md
- packages/@aws-cdk/aws-s3-assets/README.md
- packages/@aws-cdk/aws-s3-deployment/test/bucket-deployment.test.ts
- packages/@aws-cdk/aws-s3/test/bucket.test.ts
- packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py
The AWS Cloud Development Kit (CDK) provides a comprehensive suite of constructs for provisioning and managing Amazon S3 buckets and their contents. The primary construct library for this is @aws-cdk/aws-s3, which allows for the definition, configuration, and permissioning of S3 buckets. Key features include encryption, public access control, versioning, and website hosting.
For populating S3 buckets with content, the CDK offers the @aws-cdk/aws-s3-deployment module. This module enables deploying local files and directories to an S3 bucket during the CDK deployment process. It handles the packaging of local assets, uploading them to an intermediary S3 bucket, and then synchronizing them to a destination bucket using a custom CloudFormation resource. This is particularly useful for hosting static websites or distributing application assets.
Core S3 Bucket (@aws-cdk/aws-s3)
The Bucket construct in the @aws-cdk/aws-s3 module is the fundamental building block for creating S3 buckets. It provides a high-level API to configure various features of an S3 bucket.
Sources: packages/@aws-cdk/aws-s3/README.md, packages/@aws-cdk/aws-s3/lib/bucket.ts
Basic Configuration
A simple, unencrypted bucket can be created with no properties.
import * as s3 from '@aws-cdk/aws-s3';
new s3.Bucket(this, 'MyFirstBucket');
Sources: packages/@aws-cdk/aws-s3/README.md:12-14
The Bucket construct exposes numerous deploy-time attributes, such as bucketArn, bucketName, and various URL formats for accessing the bucket and its objects.
Sources: packages/@aws-cdk/aws-s3/README.md:16-33
Key Bucket Properties
The Bucket construct can be configured with several properties to control its behavior.
| Property | Type | Description |
|---|---|---|
encryption | BucketEncryption | The type of server-side encryption to use. |
versioned | boolean | Whether versioning is enabled for the bucket. Default is false. |
blockPublicAccess | BlockPublicAccess | Specifies the block public access settings for this bucket. |
removalPolicy | RemovalPolicy | The policy to apply when the bucket is removed from the stack. Default is RETAIN. |
autoDeleteObjects | boolean | If removalPolicy is DESTROY, this will automatically delete all objects in the bucket. Default is false. |
websiteIndexDocument | string | The name of the index document for website hosting. |
serverAccessLogsBucket | IBucket | The destination bucket for server access logs. |
Sources: packages/@aws-cdk/aws-s3/lib/bucket.ts, packages/@aws-cdk/aws-s3/test/bucket.test.ts
Encryption
S3 buckets can be configured with server-side encryption. The BucketEncryption enum provides several options.
The diagram below shows the relationship between the Bucket construct and its encryption properties.
| Encryption Type | Description |
|---|---|
UNENCRYPTED | Objects are not encrypted. |
S3_MANAGED | Objects are encrypted using S3-managed keys (SSE-S3). |
KMS_MANAGED | Objects are encrypted using an AWS-managed KMS key. |
KMS | Objects are encrypted using a customer-managed KMS key. An encryptionKey must be provided. |
When using BucketEncryption.KMS, you can also enable S3 Bucket Keys to reduce the cost of KMS requests.
const bucket = new Bucket(this, 'MyEncryptedBucket', {
encryption: BucketEncryption.KMS,
bucketKeyEnabled: true
});
Sources: packages/@aws-cdk/aws-s3/README.md:62-68, packages/@aws-cdk/aws-s3/test/bucket.test.ts:205-228
Permissions
Permissions can be granted to IAM principals using grant* methods like grantRead, grantWrite, and grantReadWrite. These methods modify the principal's IAM policy or the bucket's resource policy.
const lambda = new lambda.Function(this, 'Lambda', { /* ... */ });
const bucket = new Bucket(this, 'MyBucket');
// Grant the Lambda's execution role read/write access to the bucket
bucket.grantReadWrite(lambda);
This will add the necessary s3:GetObject*, s3:GetBucket*, s3:List*, s3:DeleteObject*, s3:PutObject, and s3:Abort* permissions. If the bucket is KMS-encrypted, it will also grant permissions to use the key.
Sources: packages/@aws-cdk/aws-s3/README.md:92-99, packages/@aws-cdk/aws-s3/test/bucket.test.ts:602-671
Deploying Content to S3 (@aws-cdk/aws-s3-deployment)
The @aws-cdk/aws-s3-deployment module provides the BucketDeployment construct, which allows populating an S3 bucket with local files or files from another S3 bucket.
Sources: packages/@aws-cdk/aws-s3-deployment/README.md
Deployment Architecture
The deployment process is orchestrated by a CloudFormation Custom Resource backed by an AWS Lambda function.
The following diagram illustrates the BucketDeployment workflow.
Sources
The BucketDeployment construct accepts one or more sources.
- Local Asset: A local directory or a
.zipfile. The CDK automatically packages and uploads this as an asset. - Bucket Source: A
.zipfile located in another S3 bucket.
import * as s3deploy from '@aws-cdk/aws-s3-deployment';
// Deploy from a local directory
new s3deploy.BucketDeployment(this, 'DeployWebsite', {
sources: [s3deploy.Source.asset('./website-dist')],
destinationBucket: websiteBucket,
});
// Deploy from another S3 bucket
const sourceBucket = s3.Bucket.fromBucketName(this, 'SourceBucket', 'my-source-bucket');
new s3deploy.BucketDeployment(this, 'DeployFromBucket', {
sources: [s3deploy.Source.bucket(sourceBucket, 'source.zip')],
destinationBucket: websiteBucket,
});
Sources: packages/@aws-cdk/aws-s3-deployment/README.md:33-43, packages/@aws-cdk/aws-s3-deployment/lib/source.ts:47-84
Key Deployment Properties
The BucketDeployment construct offers several options to control the deployment behavior.
| Property | Type | Description |
|---|---|---|
sources | ISource[] | The sources to deploy. |
destinationBucket | IBucket | The S3 bucket to deploy to. |
destinationKeyPrefix | string | An optional key prefix in the destination bucket. |
retainOnDelete | boolean | If false, objects will be deleted from the destination bucket when the deployment is removed. Default is true. |
prune | boolean | If true, files in the destination that are not in the source will be deleted. Default is true. |
exclude | string[] | A list of glob patterns to exclude from the deployment. |
include | string[] | A list of glob patterns to include in the deployment. |
metadata | any | User-defined metadata to set on all deployed objects. |
cacheControl | CacheControl[] | System-defined Cache-Control metadata. |
distribution | IDistribution | A CloudFront distribution to invalidate after the deployment. |
distributionPaths | string[] | The paths to invalidate in the CloudFront distribution. |
memoryLimit | number | The memory limit (in MiB) for the deployment Lambda function. Default is 128. |
Sources: packages/@aws-cdk/aws-s3-deployment/README.md, packages/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts, packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py:38-51
Pruning and Retention
prune: By default,BucketDeploymentwill delete files in the destination prefix that are not present in the source. This is equivalent to the--deleteflag inaws s3 sync. Settingprune: falsedisables this behavior.retainOnDelete: By default, when theBucketDeploymentresource is removed from the CDK stack, the files it deployed are left in the S3 bucket. SettingretainOnDelete: falsewill cause the deployment Lambda to delete the files from the destination upon resource deletion.
Sources: packages/@aws-cdk/aws-s3-deployment/README.md:52-82, 84-88, packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py:79-81, 131-132
CloudFront Invalidation
If you are using a CloudFront distribution in front of your S3 bucket, you can automatically invalidate the cache after a deployment.
import * as cloudfront from '@aws-cdk/aws-cloudfront';
import * as origins from '@aws-cdk/aws-cloudfront-origins';
const bucket = new s3.Bucket(this, 'Destination');
const distribution = new cloudfront.Distribution(this, 'Distribution', {
defaultBehavior: { origin: new origins.S3Origin(bucket) },
});
new s3deploy.BucketDeployment(this, 'DeployWithInvalidation', {
sources: [s3deploy.Source.asset('./website-dist')],
destinationBucket: bucket,
distribution,
distributionPaths: ['/images/*.png'], // Paths to invalidate
});
The deployment Lambda will make an API call to cloudfront.create_invalidation after the S3 sync is complete.
Sources: packages/@aws-cdk/aws-s3-deployment/README.md:154-168, packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py:101-102, 144-153
Bucket Notifications
The @aws-cdk/aws-s3-notifications module provides target implementations for S3 bucket event notifications. You can configure a bucket to send a notification to a Lambda function, an SQS queue, or an SNS topic when an event like s3:ObjectCreated:* occurs.
The following sequence diagram shows the flow for an object creation event triggering a Lambda function.
Configuration
To set up a notification, use the addEventNotification method on the Bucket construct.
import * as s3n from '@aws-cdk/aws-s3-notifications';
import * as lambda from '@aws-cdk/aws-lambda';
const myLambda = new lambda.Function(this, 'MyFunction', { ... });
const bucket = new s3.Bucket(this, 'MyBucket');
// Trigger the lambda for every object created
bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(myLambda));
This automatically adds the necessary permissions to the Lambda function's execution role and the bucket's resource policy to allow S3 to invoke the function.
Sources: packages/@aws-cdk/aws-s3/README.md:193-199, packages/@aws-cdk/aws-s3-notifications/lib/lambda.ts a
Summary
The AWS CDK provides a powerful and convenient way to manage S3 resources. The @aws-cdk/aws-s3 module offers a high-level Bucket construct for easy configuration of encryption, permissions, and other features. The @aws-cdk/aws-s3-deployment module simplifies the process of deploying static content to S3 buckets, integrating seamlessly with local assets and CloudFront for cache invalidation. Finally, @aws-cdk/aws-s3-notifications enables event-driven architectures by connecting S3 events to other AWS services like Lambda, SQS, and SNS. Together, these modules form a robust toolkit for object storage management within an Infrastructure as Code paradigm.