CI/CD with CDK Pipelines

Relevant source files

The following files were used as context for generating this wiki page:

• packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts
• packages/@aws-cdk/pipelines/lib/blueprint/shell-step.ts
• design/continuous-delivery.md
• packages/@aws-cdk/pipelines/test/integ.newpipeline.ts
• packages/@aws-cdk/pipelines/ORIGINAL_API.md
• packages/@aws-cdk/pipelines/lib/legacy/pipeline.ts
• packages/@aws-cdk/pipelines/test/compliance/validations.test.ts
• packages/@aws-cdk/aws-codepipeline-actions/test/pipeline.test.ts
• README.md

The @aws-cdk/pipelines module is a construct library for the AWS Cloud Development Kit (CDK) that simplifies the process of setting up and managing continuous integration and delivery (CI/CD) pipelines for CDK applications. It provides a high-level, object-oriented abstraction to define an AWS CodePipeline pipeline entirely within your CDK application's code. This allows developers to manage their application infrastructure and their delivery pipeline using the same programming language and tooling.

CDK Pipelines automates many of the complex aspects of CI/CD, including managing stack dependencies, publishing assets (like files and Docker images), and keeping the pipeline itself up-to-date as the CDK application evolves. The library offers two main APIs: a modern, flexible API centered around the CodePipeline construct, and a legacy API (CdkPipeline) which is still supported but no longer recommended for new projects.

Sources: packages/@aws-cdk/pipelines/package.json, README.md, packages/@aws-cdk/pipelines/ORIGINAL_API.md

Core Concepts

CDK Pipelines are built on several key components that work together to define the CI/CD workflow.

Sources: packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts, packages/@aws-cdk/pipelines/lib/blueprint/shell-step.ts

Pipeline Structure

The following diagram illustrates the hierarchical relationship between the main components of a CDK Pipeline.

*CodePipeline**: The root construct that represents the entire CI/CD pipeline. It orchestrates all stages and steps.

• Synth Step: A required step that synthesizes the CDK application to produce a Cloud Assembly. This is typically a ShellStep or CodeBuildStep.
• Wave: A collection of one or more Stages that are deployed in parallel. Waves are executed sequentially.
• Stage: A logical deployment environment, such as 'testing' or 'production', which contains one or more CDK stacks.
• Step: An individual action within the pipeline, such as running a script (ShellStep), executing a CodeBuild project (CodeBuildStep), or requiring a manual approval (ManualApprovalStep). Steps can be added before (pre) or after (post) a Stage or Wave.

Sources: packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts:25-50, packages/@aws-cdk/pipelines/lib/blueprint/shell-step.ts:20-30

Self-Mutation

A key feature of CDK Pipelines is self-mutation. By default, the pipeline includes an UpdatePipeline stage that automatically updates the pipeline itself if any changes are made to its definition in the CDK code (e.g., adding a new stage or action). This is controlled by the selfMutation property, which is enabled by default.

Sources: packages/@aws-cdk/pipelines/lib/legacy/pipeline.ts:119-129, packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts:133-138

Defining a Pipeline

There are two APIs for defining a pipeline: the modern CodePipeline construct and the legacy CdkPipeline construct. The modern API is more flexible and is recommended for all new projects.

Modern API (CodePipeline)

The CodePipeline construct is the centerpiece of the modern API. It requires a synth step, which defines how to build the source code and synthesize the CDK Cloud Assembly.

Here is an example of a basic pipeline that sources from GitHub and deploys a single-stack application:

// From packages/@aws-cdk/pipelines/test/integ.newpipeline.ts
const pipeline = new cdkp.CodePipeline(this, 'Pipeline', {
  pipelineName: 'MyTestPipeline',
  synth: new cdkp.ShellStep('Synth', {
    input: cdkp.CodePipelineSource.gitHub('amzn/aws-cdk-testing', 'main'),
    commands: [
      'npm ci',
      'npm run build',
      'npx cdk synth',
    ],
  }),
});

pipeline.addStage(new OneStackApp(this, 'App'));

Sources: packages/@aws-cdk/pipelines/test/integ.newpipeline.ts:26-39

Key CodePipeline Properties

Property	Type	Description
synth	Step	The step that produces the CDK Cloud Assembly. Typically a ShellStep or CodeBuildStep.
selfMutation	boolean	Whether the pipeline will update itself. Defaults to true.
crossAccountKeys	boolean	Whether to create KMS keys for cross-account deployments. Defaults to false. Set to true if deploying to other accounts.
dockerEnabledForSelfMutation	boolean	Set to true if the pipeline's self-mutation step needs to build Docker images.
codeBuildDefaults	CodeBuildOptions	Default settings for all CodeBuild projects in the pipeline, such as vpc or buildEnvironment.
codePipeline	codepipeline.Pipeline	An existing AWS CodePipeline to add stages to, allowing for customization.

Sources: packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts:74-150, packages/@aws-cdk/pipelines/ORIGINAL_API.md:13-25

Migrating from Legacy to Modern API

The legacy CdkPipeline API has been superseded by the more flexible CodePipeline API. Migration involves replacing new CdkPipeline with new CodePipeline and updating the configuration properties.

The following diagram shows the conceptual change from addApplicationStage to addStage.

Key property changes are summarized below:

Legacy API (CdkPipeline)	Modern API (CodePipeline)	Notes
sourceAction	synth.input	The source is now defined as an input to the synth step.
synthAction	synth	Replaced by a Step object, typically ShellStep.
cloudAssemblyArtifact	(removed)	Managed automatically by the pipeline.
crossAccountKeys: true	crossAccountKeys: true	Default changed from true to false.
selfMutating	selfMutation	Renamed for clarity.
addApplicationStage()	addStage()	The method for adding deployment stages was renamed.

Sources: packages/@aws-cdk/pipelines/ORIGINAL_API.md:6-50

Pipeline Structure and Flow

A typical CDK Pipeline follows a standard sequence of stages to deliver an application.

This diagram shows the default flow of a self-mutating CDK Pipeline.

Source and Synthesis

The pipeline starts by fetching source code and synthesizing the CDK app into a Cloud Assembly. This is defined in the synth step.

• CodePipelineSource: A factory class to create inputs from sources like GitHub (CodePipelineSource.gitHub(...)) or AWS CodeCommit (CodePipelineSource.codeCommit(...)).
• ShellStep: A versatile step that runs shell commands. It's commonly used for the synth action to install dependencies, compile code, and run cdk synth.

// Example of a synth step
new ShellStep('Synth', {
  input: CodePipelineSource.gitHub('OWNER/REPO', 'main'),
  commands: [
    'npm ci',
    'npm run build',
    'npx cdk synth',
  ],
});

Sources: packages/@aws-cdk/pipelines/ORIGINAL_API.md:57-66, packages/@aws-cdk/pipelines/lib/blueprint/shell-step.ts:20-29

Application Deployment

Applications are deployed by adding Stage constructs to the pipeline.

• addStage(stage): Adds a new deployment stage to the pipeline. The stages are deployed sequentially by default.
• addWave(id, options): Adds a wave, which can contain multiple stages that will be deployed in parallel.

// Deploy two stages in parallel in a 'Beta' wave
const betaWave = pipeline.addWave('Beta');
betaWave.addStage(new MyApplication(this, 'Beta-EU', { env: { region: 'eu-west-1' } }));
betaWave.addStage(new MyApplication(this, 'Beta-US', { env: { region: 'us-east-1' } }));

// Deploy a production stage after the beta wave completes
pipeline.addStage(new MyApplication(this, 'Prod'));

Sources: packages/@aws-cdk/pipelines/lib/codepipeline/codepipeline.ts:365-385

Validation and Manual Approvals

CDK Pipelines allows for adding validation actions and manual approvals to ensure the quality and safety of deployments. These are added to the pre (before deployment) or post (after deployment) properties of a stage or wave.

Scripted Validations

You can run automated tests or validation scripts using ShellStep or CodeBuildStep. A common use case is to run integration tests against a newly deployed stage, using its CloudFormation outputs.

The envFromCfnOutputs property makes it easy to consume stack outputs as environment variables in your script.

// my-app-stage.ts
export class AppWithOutput extends cdk.Stage {
  public readonly theOutput: cdk.CfnOutput;
  constructor(scope: Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);
    const stack = new cdk.Stack(this, 'Stack');
    this.theOutput = new cdk.CfnOutput(stack, 'MyOutput', { value: 'output-value' });
  }
}

// pipeline-stack.ts
const myApp = new AppWithOutput(app, 'Alpha');
pipeline.addStage(myApp, {
  post: [
    new cdkp.ShellStep('Validate', {
      commands: ['curl -Ssf $THE_OUTPUT'],
      envFromCfnOutputs: {
        THE_OUTPUT: myApp.theOutput,
      },
    }),
  ],
});

Sources: packages/@aws-cdk/pipelines/test/compliance/validations.test.ts:98-115

Manual Approvals

To pause the pipeline for a manual check, add a ManualApprovalStep. This is often done before deploying to a production environment.

const prodStage = pipeline.addStage(new MyProductionApp(this, 'Prod'));

prodStage.addPre(new cdkp.ManualApprovalStep('PromoteToProd'));

Sources: packages/@aws-cdk/pipelines/ORIGINAL_API.md:105-113, packages/@aws-cdk/pipelines/test/compliance/validations.test.ts:25-33

Advanced Configurations

Cross-Account & Cross-Region Deployments

CDK Pipelines is designed to support deployments to different AWS accounts and regions from a single pipeline.

• Cross-Account: To enable cross-account deployments, set crossAccountKeys: true on the CodePipeline construct. This creates the necessary KMS keys for artifact encryption and sharing. The roles and permissions required for the target account to access the pipeline's artifact bucket are configured automatically.
• Cross-Region: When an action or stage targets a different region, the pipeline automatically creates a replication bucket in that region and handles artifact replication.

Sources: packages/@aws-cdk/pipelines/ORIGINAL_API.md:162-174, packages/@aws-cdk/aws-codepipeline-actions/test/pipeline.test.ts:490-509

Using an Existing CodePipeline

If you need fine-grained control over the AWS CodePipeline resource, you can create one yourself and pass it to the CodePipeline construct via the codePipeline property. CDK Pipelines will then add its stages (like UpdatePipeline, Assets, and application stages) to the existing pipeline object.

const existingCodePipeline = new cp.Pipeline(this, 'CodePipeline', {
  stages: [
    {
      stageName: 'CustomSource',
      actions: [new TestGitHubAction(sourceArtifact)],
    },
  ],
});

new cdkp.CodePipeline(this, 'Cdk', {
  codePipeline: existingCodePipeline,
  synth: new cdkp.ShellStep('Synth', {
    input: cdkp.CodePipelineFileSet.fromArtifact(sourceArtifact),
    commands: ['true'],
  }),
});

Sources: packages/@aws-cdk/pipelines/test/compliance/escape-hatching.test.ts:74-84, packages/@aws-cdk/pipelines/ORIGINAL_API.md:223-236

Conclusion

CDK Pipelines provides a robust, "infrastructure-as-code" approach to continuous delivery for applications built with the AWS CDK. By defining the entire CI/CD process in a familiar programming language, it empowers developers to automate builds, tests, and deployments with high-level, reusable constructs. Its built-in support for self-mutation, asset management, and multi-account deployments makes it a powerful tool for managing the complete lifecycle of cloud applications on AWS.