Code Generation (`pulumi-codegen`)

Relevant source files

The following files were used as context for generating this wiki page:

• pkg/codegen/README.md
• pkg/codegen/go/gen.go
• pkg/codegen/python/gen.go
• pkg/codegen/nodejs/gen.go
• pkg/codegen/dotnet/gen.go
• pkg/codegen/nodejs/gen_program.go
• pkg/codegen/testing/test/program_driver.go
• pkg/codegen/testing/test/helpers.go
• pkg/cmd/pulumi/markdown/gen.go

The code generation subsystem, also known as Pulumi CrossCode, is a core component of Pulumi's architecture. Its primary purpose is to enable multi-language support for a wide variety of cloud providers by translating a language-agnostic schema definition into language-specific SDKs. This allows developers to interact with cloud resources using their preferred programming languages like Go, Python, Node.js (TypeScript), and .NET (C#).

The pkg/codegen directory contains the logic for this system. It manages three main categories of output: provider SDKs, example programs, and documentation. The foundation of this system relies on shared concepts like the Pulumi Schema and Pulumi Configuration Language (PCL).

Sources: pkg/codegen/README.md:5-12

Overall Architecture

The code generation process starts with a Pulumi Schema file (schema.json or schema.yaml), which defines the resources, functions, and types for a given provider. The language-specific generators consume this schema to produce a complete, typed SDK for the target language.

This diagram illustrates the high-level flow of SDK generation.

Language-Specific SDK Generators

Each supported language has a dedicated generator package responsible for translating the schema into idiomatic code for that language. While the implementation details vary, they follow a similar pattern:

1. Load Schema: The process begins by loading and binding the provider's schema definition.
2. Build Module Context: The generator analyzes the schema to group resources, types, and functions into modules. A modContext (or similarly named struct) is created for each module to manage its state, including names, types, and dependencies.
3. Generate Files: The generator iterates through the module contexts and generates source files for resources, functions, enums, and complex types.
4. Generate Metadata: Finally, it creates language-specific project and dependency files like go.mod, package.json, .csproj, and pyproject.toml.

Sources: pkg/codegen/go/gen.go:1938-1950, pkg/codegen/python/gen.go:1240-1254, pkg/codegen/nodejs/gen.go:1210-1223, pkg/codegen/dotnet/gen.go:1546-1558

Core Data Structure: modContext

The modContext (or pkgContext in Go) is a central data structure in each language generator. It holds all the necessary information to generate the code for a single module within the provider SDK.

This class diagram shows the typical structure of the context object used by the generators.

Go Generator

The Go generator is implemented in pkg/codegen/go/gen.go.

• Entry Point: The main function is GeneratePackage.
• Context: It uses a pkgContext struct to manage generation state for each Go package.
• File Generation: It generates separate .go files for resources, functions, enums, and input/output types. It also handles the generation of doc.go, init.go for resource registration, and utility files.
• Naming: It converts Pulumi module names (e.g., someModule) into Go package names (e.g., somemodule). It also handles potential name collisions between types and resources.
• Metadata: It creates a go.mod file if the SupportPack option is enabled in the package's Go language metadata.
• Generics: The Go generator has a Generics setting in GoPackageInfo which can be none, side-by-side, or generics-only to control the emission of generic-based SDKs.

Sources: pkg/codegen/go/gen.go:2308-2321, 2465-2495, 252-255

Python Generator

The Python generator is implemented in pkg/codegen/python/gen.go.

• Entry Point: The main function is GeneratePackage.
• Context: It uses a modContext struct.
• File Generation: It creates an __init__.py for each module, which exports the module's members. Resources, functions, and enums are generated into their own files (_enums.py, _inputs.py, outputs.py).
• Naming: It converts Pulumi module names into Python-compliant module names (snake_case). It respects moduleNameOverrides from the package info.
• Metadata: It generates setup.py and, if enabled via PyProject.Enabled, a pyproject.toml file for package metadata and dependencies.
• Input Types: A notable feature is the InputTypes setting, which can be classes or classes-and-dicts, controlling whether to generate TypedDict types for inputs.

Sources: pkg/codegen/python/gen.go:1240-1254, 30-33, 563-570

Node.js (TypeScript) Generator

The Node.js generator is implemented in pkg/codegen/nodejs/gen.go.

• Entry Point: The main function is GeneratePackage.
• Context: It uses a modContext struct.
• File Generation: It generates an index.ts file for each module that exports all its members. It also creates a types subdirectory containing input.ts, output.ts, and enums.ts.
• Naming: It uses camelCase for file and function names and PascalCase for types and resources.
• Metadata: It generates package.json with dependencies and a tsconfig.json file for compilation.

Sources: pkg/codegen/nodejs/gen.go:1210-1223, 1145-1153

.NET (C#) Generator

The .NET generator is implemented in pkg/codegen/dotnet/gen.go.

• Entry Point: The main function is GeneratePackage.
• Context: It uses a modContext struct.
• File Generation: It organizes code into namespaces. Resources, arguments (*Args), and state (*State) types are typically generated in the same file. Input and output types are placed in Inputs and Outputs sub-namespaces.
• Naming: It uses PascalCase for namespaces, classes, and properties, following C# conventions. It handles C# reserved keywords by prefixing them with @.
• Metadata: It generates a .csproj file for the project, including package references and other MSBuild properties. It also embeds a pulumi-plugin.json file.

Sources: pkg/codegen/dotnet/gen.go:1546-1558, 87-107, 1618-1627

Language-Specific Settings

Each language generator can be customized through a language-specific block in the Pulumi schema.

Language	Struct	Key Settings	Source File
Go	GoPackageInfo	importBasePath, moduleToPackage, generateResourceContainerTypes, generics	pkg/codegen/go/gen.go
Python	PackageInfo	packageName, moduleNameOverrides, requires, pyproject	pkg/codegen/python/gen.go
Node.js	NodePackageInfo	packageName, moduleToPackage, dependencies, devDependencies	pkg/codegen/nodejs/gen.go
.NET	CSharpPackageInfo	packageName, namespaces, packageReferences, projectReferences	pkg/codegen/dotnet/gen.go

Sources: pkg/codegen/go/gen.go, pkg/codegen/python/gen.go, pkg/codegen/nodejs/gen.go, pkg/codegen/dotnet/gen.go

Program Generation from PCL

Besides SDKs, the codegen system can also generate complete, runnable programs from the Pulumi Configuration Language (PCL). This is used by features like pulumi import and Pulumi AI to generate code from existing infrastructure or natural language prompts.

The process is handled by language-specific program generators, such as the one in pkg/codegen/nodejs/gen_program.go.

The core of the program generator is a generator struct that walks the linearized graph of PCL nodes and emits corresponding code for each node type (Resource, ConfigVariable, LocalVariable, OutputVariable). A key challenge is handling dependencies and asynchronous operations, which sometimes requires switching the main program entry point to be async.

Sources: pkg/codegen/nodejs/gen_program.go:41-50, 110-135

CLI Documentation Generation

The codegen system is also used internally to generate documentation for the Pulumi CLI. The pkg/cmd/pulumi/markdown/gen.go file contains logic to generate Markdown files for each CLI command.

• It uses the github.com/spf13/cobra/doc package to generate the base Markdown.
• It then post-processes the files to add YAML front matter for title, aliases, and meta descriptions, which are used by the documentation website.
• It adjusts heading levels (promoting ### to ##) to ensure a consistent document structure.

Sources: pkg/cmd/pulumi/markdown/gen.go:90-128

Testing

The codegen system is rigorously tested using a baseline-driven approach. The testing framework is located in pkg/codegen/testing/test/.

• Test Driver: The TestProgramCodegen function in program_driver.go orchestrates the tests. It iterates through a list of ProgramTest structs, each defining a test case.
• Test Case: A ProgramTest specifies a directory containing a PCL file (.pp) and any language-specific options, such as which languages to skip.
• Execution: For each test case, the driver generates code for each target language and compares the output against a "golden" baseline file stored in the repository.
• Updating Baselines: If the generated code is expected to change, developers can run the tests with the PULUMI_ACCEPT=true environment variable to automatically update the baseline files.

Sources: pkg/codegen/testing/test/program_driver.go:211-298, pkg/codegen/testing/test/helpers.go:120-143

Conclusion

The pulumi-codegen system is a sophisticated and essential part of the Pulumi platform. It abstracts away the differences between cloud provider APIs and programming languages, providing a unified and idiomatic experience for developers. By leveraging a single source of truth (the Pulumi Schema), it ensures consistency and scalability, enabling Pulumi to rapidly expand its ecosystem of supported clouds and languages. The system's ability to generate not just SDKs but also full programs and documentation highlights its versatility and central role in the developer experience.