The Pulumi Engine

The Pulumi Deployment Engine is the core component responsible for orchestrating the deployment and management of infrastructure. It takes a user's infrastructure-as-code program, compares the desired state with the current state, and determines the necessary actions to converge the two. The engine communicates with language hosts to run the user's code and with resource providers to perform CRUD (Create, Read, Update, Delete) operations on cloud resources.

The engine is one of five primary components in the Pulumi architecture, alongside State Storage Backends, Language SDKs, Resource Providers, and Package Schemas. It acts as the central coordinator, driving the entire deployment process from start to finish.

Sources: developer-docs/architecture/overview.md

Core Architecture

The Pulumi Engine sits at the center of the Pulumi ecosystem, interacting with several other components via gRPC interfaces.

This diagram illustrates the high-level interaction between the main Pulumi components during a deployment.

Pulumi CLI*: The command-line interface is the primary user entry point. It invokes the Deployment Engine to perform actions like up, preview, destroy, and import.

• Deployment Engine: The orchestrator. It interprets the user's intent, plans the deployment, and executes it.
• Language Host: A process responsible for running a Pulumi program written in a specific language (e.g., Node.js, Python, Go, .NET). It communicates resource registration requests back to the engine.
• User Program: The user's code that defines the desired infrastructure using a Pulumi SDK.
• Resource Provider: A plugin that understands how to interact with a specific cloud or service API (e.g., AWS, Azure, Kubernetes) to manage its resources.
• State Storage Backend: Where Pulumi stores the state of the managed infrastructure. This can be the Pulumi Service, a local file, or a self-managed cloud object store.

Sources: developer-docs/architecture/overview.md, proto/README.md, sdk/nodejs/tests/runtime/langhost/run.spec.ts

The Deployment Lifecycle

When a user runs a command like pulumi up, the engine initiates a deployment lifecycle to bring the actual infrastructure into the desired state defined by the code.

This sequence diagram shows the typical flow for a pulumi up command.

1. Program Execution & Resource Registration

The engine starts the appropriate language host for the project. The language host then executes the user's Pulumi program. As the program runs, it uses the Pulumi SDK to declare resources. Each resource declaration results in a RegisterResource gRPC call to the engine.

Sources: sdk/nodejs/tests/runtime/langhost/run.spec.ts:1000-1070

2. Step Generation

Once the program completes, the engine has a complete view of the desired state. It compares this desired state against the previously saved state from the backend. This comparison, known as a "diff", produces a series of steps required to converge the states.

The possible step operations include:

• Create: A new resource needs to be provisioned.
• Update: An existing resource has changed properties and needs to be modified.
• Delete: A resource exists in the old state but not in the new one and should be removed.
• Replace: An existing resource has an immutable property change, requiring the old resource to be deleted and a new one created.
• Same: The resource has not changed.
• Read: An existing resource is being read or imported into the stack.

3. Step Execution

The engine's step executor processes the generated steps in an order determined by the resource dependency graph. For each step, it makes the corresponding gRPC call to the appropriate resource provider. For example, a Create step results in a Create call to the provider plugin. The provider then interacts with the target cloud API to perform the operation.

Sources: sdk/nodejs/tests/runtime/langhost/run.spec.ts:1010-1070, developer-docs/architecture/import.md

Resource Import

The engine supports bringing existing, unmanaged cloud resources under Pulumi's management. This is handled through two primary mechanisms.

Sources: developer-docs/architecture/import.md

import Resource Option

A user can add an import: "resource-id" option to a resource definition in their code. During the next pulumi up, the engine will attempt to adopt the existing resource instead of creating a new one.

The process requires that the resource definition in the code accurately describes the state of the resource being imported to avoid accidental changes.

This flowchart shows the process for the import resource option.

pulumi import Command

The pulumi import command is a more direct way to import resources. It takes a resource type, name, and ID, imports the resource into the state, and generates the corresponding code definition for the user to add to their program.

This process is more automated, as the engine attempts to derive the minimal set of inputs required to represent the resource's current state.

This flowchart shows the process for the pulumi import command.

Interacting with the Engine

While the Pulumi CLI is the most common way to interact with the engine, it's not the only way.

The Automation API provides a way to drive the Pulumi engine programmatically, encapsulating the functionality of the CLI in a library available for languages like Go, Node.js, Python, and .NET.

Sources: sdk/go/auto/README.md

Code Generation and Schemas

The engine's ability to understand resources and providers is based on a well-defined schema. Each Pulumi provider package contains a schema.json file that describes its resources, their properties, and functions.

The pulumi.json schema defines the structure for these provider schemas, including:

• name, version, description, license, etc.
• resources: A map of resource tokens to their specifications (resourceSpec).
• functions: A map of function tokens to their specifications (functionSpec).
• config: Configuration variables for the package.
• types: Definitions for complex object types.

The language-specific code generators (pkg/codegen/{language}/gen.go) use this schema to generate the strongly-typed SDKs that developers use in their Pulumi programs. For example, pkg/codegen/python/gen.go contains the logic to generate Python classes for resources and input/output types from the schema. The genResourceModule function in these generators creates the code that registers resources with the Pulumi runtime, which in turn communicates with the engine.

Sources: pkg/codegen/schema/pulumi.json, pkg/codegen/python/gen.go:2140-2180, pkg/codegen/dotnet/gen.go:1640-1678, pkg/codegen/go/gen.go:1950-2015, pkg/codegen/nodejs/gen.go:1380-1444

Summary

The Pulumi Engine is the heart of the Pulumi platform. It acts as a sophisticated desired-state engine that translates declarative infrastructure-as-code programs into a series of concrete actions. By orchestrating language hosts and resource providers, it reliably and predictably manages the lifecycle of cloud infrastructure, from creation and updates to importation and deletion. Its architecture allows for a flexible and extensible system that supports multiple languages and a vast ecosystem of cloud providers.