Caching and Output Management
Relevant source files
The following files were used as context for generating this wiki page:
Cloud Custodian manages resource data through an intelligent caching system and a flexible output architecture. The caching system minimizes API calls to cloud providers by persisting resource state, while the output management system captures, formats, and uploads policy execution results, logs, and metrics to various destinations.
Structured outputs provide visibility into which resources matched specific policies, allowing for unified reporting across multi-cloud environments. The system supports local filesystem storage, cloud-native object storage (like Amazon S3), and various CLI formats for developer feedback.
Sources: README.md:47-68, tests/test_output.py:20-40
Caching Architecture
Caching reduces the number of repetitive API requests by storing resource information locally. This is critical for large fleets where API rate limits might otherwise be exceeded.
Cache Mechanisms
- API Call Minimization: The system uses intelligent cache lookups to avoid redundant calls to cloud provider endpoints.
- Persistence: Caching mechanisms allow policies to run against previously retrieved resource sets within a defined time window.
Sources: README.md:65, c7n/cache.py
Output Management System
Output management handles the lifecycle of data generated during a policy run. This includes the match results (resources that triggered the policy), execution logs, and performance metrics.
Core Output Components
The system utilizes specialized classes to handle different destination types and data formats:
| Component | Description |
|---|---|
DirectoryOutput | Manages local filesystem outputs using a file:// URL scheme. |
S3Output | Handles uploads to Amazon S3, supporting custom key prefixes and encryption. |
Metrics | Abstract base for capturing policy performance and match counts. |
LogFile | Intercepts standard logging to save policy-specific execution logs. |
Sources: tests/test_output.py:20-55, tools/c7n_left/c7n_left/output.py:40-60
Execution Data Flow
The following diagram illustrates how resource data transitions from cloud providers through the filtering engine into the output management system.
The flow shows that resource retrieval first checks the cache to optimize performance before filtering matches and dispatching to configured outputs.
Sources: README.md:47-68, tests/test_output.py:50-100
Object Storage and S3 Integration
S3Output manages the transmission of result files to Amazon S3. It allows dynamic path construction using templates.
Dynamic Path Formatting
Output paths can include variables such as {region}, {account_id}, {policy_name}, and {now}. For example, a path configured as s3://prefix/{region}/{account_id}/{policy_name}/{now:%Y} resolves to a structured hierarchy based on the execution context.
Sources: tests/test_output.py:75-84
Server-Side Encryption (SSE)
S3Output supports multiple encryption configurations via URL parameters:
- AES256: Default encryption managed by S3.
- AWS KMS: Uses AWS-managed KMS keys via
sse=aws:kms. - Customer KMS: Specifies a specific key ID via
kms-key=alias/name.
Sources: tests/test_output.py:165-195
ShiftLeft (IaC) Output Formats
For infrastructure-as-code (IaC) scanning, the c7n-left module provides specialized output formats for CI/CD pipelines and developer environments.
Supported CLI Formats
- CLI: Uses the
richlibrary to provide formatted, color-coded failures directly in the terminal, including source code snippets of the failing resource. - JSON/JSONGraph: Generates machine-readable results or full graph representations for further analysis.
- JUnit: Creates XML reports compatible with most CI/CD test viewers.
- GitHub/GitLab: Formats output as action annotations or SAST reports for native platform integration.
Sources: tools/c7n_left/c7n_left/output.py:65-135, tools/c7n_left/c7n_left/output.py:465-510
Summary Reporting
The system provides summaries by policy or by resource type, showing compliance counts and failure rates.
The hierarchy demonstrates how specialized summary classes extend the base output functionality to track metrics across different dimensions.
Sources: tools/c7n_left/c7n_left/output.py:165-300
Documentation and Policy Data Gathering
The c7n-autodoc utility gathers policy information for automated documentation generation. It crawls a local directory for YAML policy files, groups them by resource type and category (e.g., 'Cost Controls' or 'Security'), and generates HTML reports. These reports are then uploaded to S3 with public-read ACLs to serve as static documentation.
Sources: tools/sandbox/c7n_autodoc/c7n-autodoc.py:70-110, tools/sandbox/c7n_autodoc/README.md:10-25
Metrics Collection
The Metrics system captures numeric data points during policy execution. Policies buffer these metrics and then flush them to providers like CloudWatch.
Metric Data Points
Standard metrics include:
- Resource Count: Number of resources matching a policy.
- Execution Time: Duration of the policy run.
- Action Success/Failure: Metrics related to specific actions performed on resources.
Sources: tests/test_output.py:20-35, c7n/output.py
Caching and output management ensure that Cloud Custodian remains performant at scale while providing detailed, actionable insights into cloud resource compliance across diverse storage and reporting backends.