gRPC Protocol
Relevant source files
The following files were used as context for generating this wiki page:
The Pulumi architecture is built upon a system of communicating components that interact via gRPC. The interfaces for this communication are defined using Protocol Buffers (Protobuf), which serve as the single source of truth for the types, messages, and services exchanged between the Pulumi engine, language runtimes, and resource providers. This design allows for a pluggable system where different languages and cloud providers can be supported seamlessly.
The core components communicate over three primary gRPC services: ResourceMonitor, LanguageRuntime, and ResourceProvider. The Protobuf definitions for these services are located in the proto/ directory and are used to generate client and server stubs for Go, Node.js, and Python.
Sources: proto/README.md
Architecture Overview
Pulumi's operation involves three main processes: the Pulumi Engine, the Language Host, and one or more Resource Provider plugins. They communicate via gRPC services hosted by each component.
This diagram shows the three main components and the gRPC services they use to communicate.
- Pulumi Engine: The central orchestrator, responsible for driving deployments. It communicates with the Language Host to run the user's program and with Provider Plugins to manage resources. It exposes the
ResourceMonitorservice. - Language Host: A process responsible for running a Pulumi program written in a specific language (e.g., Node.js, Python, Go). It hosts the
LanguageRuntimeservice and makes calls to the Engine'sResourceMonitorservice to register resources. - Provider Plugin: A process that understands how to interact with a specific cloud API (e.g., AWS, Azure, Kubernetes). It hosts the
ResourceProviderservice, which the Engine calls to perform CRUD operations on resources.
Sources: proto/pulumi/language.proto, proto/pulumi/resource.proto, proto/pulumi/provider.proto
Typical pulumi up Flow
The following sequence diagram illustrates the interactions between components during a pulumi up operation.
This diagram shows the sequence of gRPC calls made to execute a Pulumi program and create a resource.
Sources: proto/pulumi/language.proto, proto/pulumi/resource.proto, proto/pulumi/provider.proto
Core Services
ResourceMonitor
The ResourceMonitor service is hosted by the Pulumi Engine. Language runtimes connect to this service to perform resource-related operations.
Sources: proto/pulumi/resource.proto
| RPC Method | Description | Request | Response |
|---|---|---|---|
SupportsFeature | Checks if the engine supports a specific feature. | SupportsFeatureRequest | SupportsFeatureResponse |
Invoke | Dynamically invokes a provider function. | ResourceInvokeRequest | InvokeResponse |
Call | Dynamically calls a provider method. | ResourceCallRequest | CallResponse |
ReadResource | Reads the state of an existing resource. | ReadResourceRequest | ReadResourceResponse |
RegisterResource | Registers a new resource with the engine. | RegisterResourceRequest | RegisterResourceResponse |
RegisterResourceOutputs | Registers the outputs for a resource that has been created. | RegisterResourceOutputsRequest | google.protobuf.Empty |
Key Message: RegisterResourceRequest
This message is sent from a language runtime to the engine to declare a resource.
| Field | Type | Description |
|---|---|---|
type | string | The resource's type token (e.g., aws:s3/bucket:Bucket). |
name | string | The logical name of the resource in the program. |
parent | string | The URN of the parent resource, if any. |
custom | bool | true if this is a custom resource managed by a provider. |
object | google.protobuf.Struct | A struct containing the resource's input properties. |
dependencies | repeated string | A list of URNs for resources this resource explicitly depends on. |
provider | string | A reference to a specific provider instance to use for this resource. |
propertyDependencies | map<string, PropertyDependencies> | Specifies fine-grained dependencies between properties. |
version | string | The version of the provider plugin to use. |
aliases | repeated Alias | A list of old URNs that this resource may have had. |
Sources: proto/pulumi/resource.proto:141-218
LanguageRuntime
The LanguageRuntime service is implemented by each language host (e.g., pulumi-language-nodejs). The Pulumi Engine calls this service to execute a Pulumi program.
Sources: proto/pulumi/language.proto
| RPC Method | Description | Request | Response |
|---|---|---|---|
Run | Executes a Pulumi program. | RunRequest | RunResponse |
GetRequiredPlugins | Gets the set of plugins required by the program. | GetRequiredPluginsRequest | GetRequiredPluginsResponse |
InstallDependencies | Installs program dependencies. | InstallDependenciesRequest | InstallDependenciesResponse |
GenerateProgram | Generates a Pulumi program from a PCL source. | GenerateProgramRequest | GenerateProgramResponse |
GenerateProject | Generates a Pulumi project from a PCL source. | GenerateProjectRequest | GenerateProjectResponse |
GeneratePackage | Generates a Pulumi package from a schema. | GeneratePackageRequest | GeneratePackageResponse |
Key Message: RunRequest
This message contains all the information a language runtime needs to execute a program.
| Field | Type | Description |
|---|---|---|
project | string | The project name. |
stack | string | The stack name. |
pwd | string | The program's working directory. |
program | string | The entry point of the program. |
args | repeated string | Command-line arguments for the program. |
config | map<string, string> | Configuration variables for the stack. |
dryRun | bool | true if this is a preview, false for an update. |
monitor_address | string | The gRPC address of the engine's ResourceMonitor service. |
Sources: proto/pulumi/language.proto:115-144
ResourceProvider
The ResourceProvider service is implemented by each resource provider plugin. The Pulumi Engine calls this service to manage the lifecycle of physical resources.
Sources: proto/pulumi/provider.proto
| RPC Method | Description | Request | Response |
|---|---|---|---|
GetSchema | Returns the provider's schema. | GetSchemaRequest | GetSchemaResponse |
CheckConfig | Validates the provider's configuration. | CheckRequest | CheckResponse |
DiffConfig | Checks for differences in the provider's configuration. | DiffRequest | DiffResponse |
Configure | Configures the provider with credentials and other settings. | ConfigureRequest | ConfigureResponse |
Invoke | Invokes a provider function (data source). | InvokeRequest | InvokeResponse |
Check | Validates the inputs for a resource before creation or update. | CheckRequest | CheckResponse |
Diff | Determines what changes a resource update will cause. | DiffRequest | DiffResponse |
Create | Creates a new physical resource. | CreateRequest | CreateResponse |
Read | Reads the state of an existing physical resource. | ReadRequest | ReadResponse |
Update | Updates an existing physical resource. | UpdateRequest | UpdateResponse |
Delete | Deletes a physical resource. | DeleteRequest | google.protobuf.Empty |
Construct | Creates a new component resource. | ConstructRequest | ConstructResponse |
Call | Calls a method on a resource. | CallRequest | CallResponse |
An example implementation of a ResourceProvider server can be found in the NodeJS dynamic provider.
Sources: sdk/nodejs/cmd/dynamic-provider/index.ts:114-411
Code Generation
The gRPC client and server stubs are generated from the .proto files using protoc and language-specific plugins. The proto/generate.sh script automates this process for Go, Node.js (JavaScript/TypeScript), and Python.
The generated code is committed to the repository, ensuring that developers do not need to run the generation script unless they modify the .proto definitions.
Sources: proto/README.md
The generation script performs several key steps:
- Builds a Docker container with all necessary tools (
pulumi/pulumi-build-container). - Runs
protocinside the container for each target language. - For JavaScript, it uses
sedto patch the generated code to avoid issues with global object mutation. - For Python, it uses
sedto fix import paths to be relative, making them compatible with Python 3 modules. It also modifies the generated.pyifiles to supportaioand remove abstract method decorators.
Sources: proto/generate.sh
Conclusion
The gRPC-based protocol is the backbone of Pulumi's architecture, enabling loose coupling between the engine, language runtimes, and resource providers. By defining service contracts with Protobuf, Pulumi can support multiple languages and a vast ecosystem of cloud providers in a consistent and extensible manner. This pluggable design is fundamental to Pulumi's "Infrastructure as Code in any Programming Language" philosophy.