Integration Testing Framework
Relevant source files
The following files were used as context for generating this wiki page:
The AWS CDK integration testing framework provides the tools and processes to validate the functionality of CDK components by deploying them as real AWS stacks. These tests are modeled as CDK applications that, upon successful deployment, have their synthesized CloudFormation templates "locked" into an expected state. During subsequent builds, test apps are only synthesized and compared against these locked files to detect regressions.
This framework supports several types of tests, including those for the CDK CLI, individual construct libraries, and cdk init templates. The tests can be executed in different modes, such as during local development, in a CI/CD pipeline against build artifacts, or as canaries against the latest public release.
Sources: tools/cdk-integ-tools/README.md, packages/aws-cdk/CONTRIBUTING.md:3-5
Core Concepts
The integration testing workflow is centered around two main scenarios: updating tests during development and asserting correctness during the build process.
- Development (
cdk-integ): A developer runscdk-integto deploy a test stack. If the deployment is successful, the tool synthesizes the CloudFormation template and saves it as atest/integ.*.expected.jsonfile. This "locks" the expected output. - Build (
cdk-integ-assert): In a build environment,cdk-integ-assertis run. It synthesizes the test stacks and compares the output to the corresponding.expected.jsonfiles. The test fails if the templates differ.
Sources: tools/cdk-integ-tools/README.md:10-24
This diagram illustrates the dual workflow of the integration testing process.
Test File Convention
Integration tests are defined in files matching the pattern integ.*.ts located within a test/ directory. Each of these files should represent a CDK app that defines a single stack. For multi-stack applications, special configuration is required.
Sources: tools/cdk-integ-tools/README.md:10-12, tools/cdk-integ-tools/lib/integ-helpers.ts:271-278
Tooling
The framework is powered by the cdk-integ-tools package, which provides two key command-line tools. These are typically added as scripts to a package's package.json.
Sources: tools/cdk-integ-tools/package.json:10-13, tools/cdk-integ-tools/README.md:13-15
| Tool | Description | Typical Usage |
|---|---|---|
cdk-integ | Deploys integration test stacks to an AWS environment. On success, it updates the integ.*.expected.json file with the newly synthesized template. | npm run integ |
cdk-integ-assert | Synthesizes all integ.*.js apps and compares their output against the corresponding .expected.json files. It does not perform any deployments and is used for builds. | npm test |
Sources: tools/cdk-integ-tools/README.md:16-29
Test Execution Modes
Integration tests are designed to run in three distinct environments, each sourcing the CDK framework and CLI from a different location. This is managed by wrapper scripts that prepare the environment before executing the actual test script.
- Against Repository (
run-against-repo): Uses the CLI and libraries directly from the local source repository. This is the standard mode for developers. - Against Distribution (
run-against-dist): Uses a specific version of the CLI and libraries from a set of candidate NPM packages (build artifacts). This is used for integration tests in the CI/CD pipeline. - Against Release (
run-against-release): Uses the latest CLI and libraries from public package managers, simulating a real user's environment. This is used for canary testing.
Sources: packages/aws-cdk/CONTRIBUTING.md:7-19
The following diagram shows the different sources for each execution mode.
Test Structure and Helpers
The cdk-integ-tools package provides helper classes to discover, manage, and execute integration tests.
Sources: tools/cdk-integ-tools/lib/integ-helpers.ts
Core Classes
The IntegrationTests class discovers tests, while the IntegrationTest class represents a single test case and provides methods for synthesis and CLI invocation.
Test Configuration with Pragmas
Tests can be configured using special comments, known as "pragmas," within the test source file. These pragmas control stack selection and context settings.
| Pragma Syntax | Description - |
|---|---|
/// !cdk-integ STACK_NAME ... | Specifies which stack(s) to select in a multi-stack application. Wildcards are supported. Any word not prefixed with pragma: is considered a stack name. - |
/// !cdk-integ pragma:set-context:KEY=VALUE | Sets a context key-value pair for the synthesis. This is useful for enabling feature flags or providing other runtime context without modifying cdk.json. |
Sources: tools/cdk-integ-tools/lib/integ-helpers.ts:121-134, 303-322
Types of Integration Tests
CLI Integration Tests
These tests exercise the aws-cdk command-line tool by deploying actual stacks. They are divided into two categories:
- Functional Tests: Validate the correct deployment of stacks and actions performed by the current version of the CLI.
- Regression Tests: Ensure that changes to the CLI do not break functionality for users of older framework versions. This is achieved by running the functional tests from a previous release against the new CLI code.
The regression testing process is complex, involving multiple versions of the framework and CLI.
Sources: packages/aws-cdk/CONTRIBUTING.md:52-87
This diagram outlines the CLI regression test flow.
Construct Library Tests
Individual construct library packages (e.g., @aws-cdk/aws-s3, @aws-cdk/aws-iam) contain their own integration tests. These follow the standard integ.*.ts convention and use the cdk-integ and cdk-integ-assert tools via npm scripts. This allows each module to independently verify the behavior of its constructs.
Sources: packages/@aws-cdk/example-construct-library/README.md:58-61, packages/@aws-cdk/aws-iam/package.json:34, packages/@aws-cdk/aws-s3-assets/package.json:33
Init Template Tests
These tests validate the project templates used by the cdk init command. They ensure that a new project created with a given template (e.g., sample-app for typescript) can be initialized and compiled successfully. These tests are run against the distribution artifacts.
Sources: packages/aws-cdk/CONTRIBUTING.md:134-143, packages/aws-cdk/package.json:26
Summary
The integration testing framework is a critical component for maintaining the quality and stability of the AWS CDK. By deploying real resources and performing snapshot testing on the resulting CloudFormation templates, it provides a high degree of confidence that changes to the CLI and construct libraries do not introduce regressions. The framework's flexible execution modes and specialized tooling support a robust testing strategy that covers developer workflows, CI/CD pipelines, and post-release verification.
Sources: README.md:1-6, packages/aws-cdk/CONTRIBUTING.md:3-5