Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
- pkg/codegen/testing/test/helpers.go
- pkg/codegen/testing/test/program_driver.go
- pkg/codegen/testing/test/sdk_driver.go
- pkg/testing/integration/program.go
- sdk/nodejs/tests/automation/localWorkspace.spec.ts
- sdk/nodejs/tests/runtime/langhost/run.spec.ts
- tests/integration/steps/steps_test.go
- pkg/cmd/pulumi/packagecmd/package_publish_test.go
- pkg/codegen/schema/docs_test.go
- pkg/codegen/schema/schema_test.go
The Pulumi repository employs a multi-layered testing strategy to ensure the correctness and stability of its core components, including the CLI, code generation, language runtimes, and Automation API. This strategy encompasses everything from low-level unit tests to full end-to-end integration tests that simulate real-world user workflows.
The primary testing frameworks and approaches include high-level integration tests (ProgramTest), code generation tests for both SDKs and PCL programs, language-specific runtime and host tests, and programmatic testing of the Automation API. These are supported by a suite of helper utilities and a "golden file" or "snapshot" testing pattern to manage expected outputs.
Integration Testing (ProgramTest)
The ProgramTest framework, defined in pkg/testing/integration/program.go, is the cornerstone of Pulumi's integration testing. It orchestrates a full lifecycle of a Pulumi program, including initialization, configuration, deployment, and destruction. This allows for end-to-end validation of a program's behavior.
Sources: pkg/testing/integration/program.go
ProgramTestOptions
The behavior of a ProgramTest is configured via the ProgramTestOptions struct. This struct provides extensive control over the test environment and execution flow.
| Option | Description |
|---|---|
Dir | The program directory to test. |
Dependencies | An array of NPM packages that must be linked for the test to run. |
Config / Secrets | Maps of configuration values and secrets to set for the stack. |
EditDirs | A list of subsequent edits to apply to the program, each triggering a new deployment. |
ExtraRuntimeValidation | A callback function for performing custom validation on the stack's state after a deployment. |
ExpectFailure | A boolean indicating that the test step is expected to fail. |
Quick | A boolean to skip non-essential steps like previews and stack export/import for faster runs. |
LocalProviders | Specifies local provider plugins to be used in the test. |
CloudURL | Overrides the default Pulumi Service API endpoint, allowing tests against local backends. |
Sources: pkg/testing/integration/program.go:103-294
Test Lifecycle Flow
A typical ProgramTest executes a sequence of Pulumi commands to simulate a user's workflow. The EditDirs feature is particularly powerful, allowing tests to validate multi-step deployments and resource updates.
The following diagram illustrates the lifecycle of a ProgramTest with one edit step.
This diagram shows the flow of a ProgramTest, including initialization, an initial update, an edit, and final destruction.
Sources: pkg/testing/integration/program.go:618-675, tests/integration/steps/steps_test.go:50-98
An example of a multi-step test can be found in tests/integration/steps/steps_test.go, which validates resource creation, updates, and deletion across multiple pulumi up invocations.
Sources: tests/integration/steps/steps_test.go
Code Generation Testing
A significant portion of the testing strategy is dedicated to verifying the correctness of code generation from Pulumi schemas. This is split into two main categories: SDK generation and PCL (Pulumi Command Language) program generation.
SDK Generation Testing
SDK generation tests ensure that given a Pulumi schema, the generated SDK for a specific language is correct and compilable.
- Framework: The
TestSDKCodegenfunction andSDKTeststruct inpkg/codegen/testing/test/sdk_driver.godrive these tests. - Process: For each test case, it generates package files from a schema and compares them against a "baseline" or "golden file" snapshot.
- Golden Files: The
RewriteFilesWhenPulumiAcceptfunction allows updating these baseline files whenPULUMI_ACCEPT=trueis set. This is a common pattern for managing snapshot tests. - Checks: After generation, checks like compilation (
go/compile,dotnet/compile, etc.) can be run on the generated code.
Sources: pkg/codegen/testing/test/sdk_driver.go:20-305, pkg/codegen/testing/test/helpers.go:158-191
This diagram illustrates the process of an SDK codegen test, including baseline comparison and optional compilation checks.
Sources: pkg/codegen/testing/test/sdk_driver.go:308-375
PCL Program Generation Testing
These tests validate the transpilation of PCL (.pp files) into high-level programming languages like TypeScript, Python, Go, and C#.
- Framework:
TestProgramCodegeninpkg/codegen/testing/test/program_driver.goorchestrates these tests. - Process: It parses a PCL program, binds it to schemas, and then uses a language-specific generator (
GenProgram) to produce the target code. - Validation: The generated code is compared against an expected output file. If
PULUMI_ACCEPTis set, this file is updated. ACheckProgramOutputfunction can then run further language-specific validation, such as compilation.
Sources: pkg/codegen/testing/test/program_driver.go:415-555
Language Host and Runtime Testing
At a lower level, Pulumi tests the language hosts and runtimes themselves. The Node.js runtime tests in sdk/nodejs/tests/runtime/langhost/run.spec.ts are a prime example. These tests operate by mocking the Pulumi engine's gRPC server, allowing for fine-grained control and assertions over the communication between the language host and the engine.
Mock Engine and RPC Simulation
createMockEngineAsync: This function sets up a mock gRPC server that implements theResourceMonitorandEngineservices. Test cases provide callbacks for RPCs likeregisterResource,invoke, andreadResource.mockRun: This function initiates a test run by calling theLanguageRuntime.RunRPC on the language host, which in turn communicates with the mock engine.
This setup allows tests to assert that resources are registered with the correct properties, dependencies, and providers, without needing to provision any real infrastructure.
Sources: sdk/nodejs/tests/runtime/langhost/run.spec.ts:1403-1463
Test Case Structure
The tests are structured as a series of cases, each defining a small Pulumi program and a set of expectations and RPC mocks.
| Test Case Example | Description |
|---|---|
one_resource | A program that allocates a single resource. The test asserts registerResource is called once with the correct type and name. |
resource_thens | A program with two resources where one depends on an output of the other. The test validates dependency tracking and property propagation. |
invoke | A program that calls a function via pulumi.runtime.invoke. The test mocks the invoke RPC and validates its inputs and outputs. |
unhandled_error | A program that throws an unhandled exception. The test asserts that the language host reports the error and exits correctly. |
read_resource | A program that uses getResource. The test mocks the readResource RPC and validates its behavior. |
Sources: sdk/nodejs/tests/runtime/langhost/run.spec.ts:143-152, 257-299, 360-379, 401-423, 514-536
The following sequence diagram shows the interaction during a simple resource registration test.
This diagram shows the sequence of operations in a language runtime test, involving the test runner, a mock gRPC engine, and the language host executing a user program.
Sources: sdk/nodejs/tests/runtime/langhost/run.spec.ts:1466-1571
Automation API Testing
The Automation API, which allows for programmatic control of Pulumi, is tested in files like sdk/nodejs/tests/automation/localWorkspace.spec.ts. These tests validate the functionality of LocalWorkspace and Stack objects.
Key Tested Features
- Workspace and Stack Management: Creating, selecting, and removing stacks (
LocalWorkspace.createStack,Stack.select, etc.). - Configuration: Setting and getting configuration values, including secrets (
stack.setConfig,stack.getAllConfig). - Lifecycle Operations: Running
up,preview,refresh, anddestroyprogrammatically and inspecting their results (UpResult,PreviewResult). - Inline and Local Programs: Testing stacks that are defined both as inline functions and as on-disk projects.
- Backend Support: Tests are designed to run against both the local file backend and the Pulumi Cloud service, often determined by the
PULUMI_ACCESS_TOKENenvironment variable.
Sources: sdk/nodejs/tests/automation/localWorkspace.spec.ts:83-90, 203-242, 532-604
Automation API Test Flow
The diagram below shows a typical test flow for the Automation API, where a test creates a stack, runs an update, and then cleans up.
This flowchart depicts a standard Automation API test, from stack creation to destruction and validation of outputs.
Sources: sdk/nodejs/tests/automation/localWorkspace.spec.ts:532-570
Conclusion
The Pulumi testing strategy is comprehensive, covering multiple layers of the system. Integration tests (ProgramTest) provide high-level, end-to-end validation of user workflows. Code generation tests ensure that SDKs and PCL programs are correctly produced for all supported languages. Lower-level runtime tests for language hosts use mocking to isolate and verify the gRPC-based communication with the Pulumi engine. Finally, dedicated tests for the Automation API confirm that programmatic control over Pulumi stacks and workspaces functions as expected. This layered approach ensures high confidence in the reliability and correctness of the entire Pulumi platform.