Google Cloud Platform (GCP) Provider
Relevant source files
The following files were used as context for generating this wiki page:
The Google Cloud Platform (GCP) Provider enables Cloud Custodian to manage GCP resources through a unified rules engine. It allows users to define policies in YAML to enforce security compliance, manage tags, and optimize costs across GCP projects.
The provider integrates with the Google Cloud SDK and API discovery services to interact with resources like App Engine, Compute Engine, and Cloud Functions. It supports multi-project management and intelligent caching to minimize API calls while providing dry-run capabilities for policy validation.
Sources: README.md:14-23, tools/c7n_gcp/pyproject.toml:1-12
Core Architecture
The GCP Provider architecture relies on a specialized resource management hierarchy and the Google Discovery API. Each GCP resource is registered within the c7n_gcp namespace and defines its interaction logic via TypeInfo and ResourceManager classes.
Resource Managers
Cloud Custodian utilizes specific manager classes to handle resource lifecycles:
- QueryResourceManager: Handles top-level resources that are queried directly from an API endpoint.
- ChildResourceManager: Manages sub-resources that depend on a parent (e.g., an App Engine Version belongs to a Service).
- TypeInfo: A metadata class within each resource that defines the API service, version, component name, and required IAM permissions.
The diagram shows the relationship between core Custodian managers and the GCP-specific implementations.
Sources: tools/c7n_gcp/c7n_gcp/resources/appengine.py:11-54, tools/c7n_sphinxext/c7n_sphinxext/docgen.py:100-115
Resource Definition Logic
Resources are defined by mapping GCP API attributes to Custodian identifiers. For example, the app-engine resource maps to the appengine.googleapis.com/Application asset type and requires the appengine.applications.get permission.
App Engine Resource Example
| Attribute | Value |
|---|---|
| Service | appengine |
| API Version | v1 |
| Component | apps |
| Resource ID | id |
| Permissions | appengine.applications.get |
Sources: tools/c7n_gcp/c7n_gcp/resources/appengine.py:12-32
Child Resource Mapping
Child resources use parent_spec to define the relationship with their parent. This allows the provider to iterate through parent instances to discover all associated child instances, such as finding all firewall ingress rules for a specific App Engine application.
Sources: tools/c7n_gcp/c7n_gcp/resources/appengine.py:117-133
Testing and Functional Development
The provider employs a "Flight Data" recording pattern for testing, which allows developers to run tests without live GCP credentials or active API access.
Flight Data Mechanism
- Recording Mode: Triggered by setting
C7N_FUNCTIONAL=yes. It makes real API calls and stores the HTTP headers and bodies as JSON files. - Playback Mode: The default mode. It uses the recorded JSON files to simulate API responses.
The flowchart illustrates the dual-mode testing strategy used to ensure API stability and test reproducibility.
Sources: tools/c7n_gcp/development.md:123-149
Terraform Integration
For complex infrastructure, the provider uses the @terraform decorator. In recording mode, this automatically deploys real GCP resources via Terraform before running tests and generates tf_resources.json to store metadata for the playback mode.
Sources: tools/c7n_gcp/development.md:167-185, tools/c7n_gcp/tests/terraform/function_labels/main.tf:1-12
System Dependencies
The GCP provider relies on several key Google client libraries and management tools.
| Dependency | Purpose |
|---|---|
google-api-python-client | Core interaction with Discovery-based APIs. |
google-auth | Handles Application Default Credentials (ADC) and OAuth2. |
google-cloud-storage | Interaction with GCS buckets for serverless deployment. |
pyrate-limiter | Manages API request pacing to avoid GCP quota exhaustion. |
uv | Manages the workspace and package synchronization. |
Sources: tools/c7n_gcp/pyproject.toml:9-16, tools/c7n_gcp/development.md:27-45
Serverless Execution Modes
GCP policies can execute in different modes:
- Pull Mode: The standard mode where Custodian runs locally or on a schedule and queries the GCP API directly.
- Cloud Functions (Event-driven): Policies can be deployed as serverless functions that respond to Audit Logs or Pub/Sub events.
When deploying functions, the mu.py module manages the packaging of the policy and its deployment to the target GCP project. This includes creating the necessary IAM roles and service account bindings as defined in Terraform configurations.
Sources: tools/c7n_sphinxext/c7n_sphinxext/docgen.py:146-159, tools/c7n_gcp/tests/terraform/function_labels/main.tf:23-45
The GCP Provider serves as the bridge between Cloud Custodian's generic policy language and the specific requirements of the Google Cloud API ecosystem, ensuring consistent governance through robust resource management and testing frameworks.