Platform Dataset Exports
Relevant source files
The following files were used as context for generating this wiki page:
Platform Dataset Exports enable the extraction of large-scale governance data from the Stacklet Platform. This system bypasses the limitations of standard GraphQL pagination by generating CSV files server-side.
The service provides a more efficient mechanism for exporting thousands of records, such as resources, policies, and execution results, compared to manual paging through GraphQL connections. Users define the desired data structure through column mappings and optional filters. The system processes these requests asynchronously, providing a dataset ID for tracking and eventual download.
Sources: stacklet/mcp/platform/dataset_info.md:1-10, stacklet/mcp/platform/tools.py:126-140
Architecture and Data Flow
The export system operates as an asynchronous workflow between the MCP Server and the Stacklet Platform GraphQL API.
Export Workflow
- The user initiates an export by specifying a
connection_fieldand a list ofcolumns. - The MCP Server sends a
Q_START_EXPORTmutation to the Platform API. - The Platform API returns a unique
dataset_id. - The system either waits for completion (if a timeout is provided) or returns the ID immediately.
- Users monitor progress via the lookup tool, which polls the Platform's
nodequery until the export reaches a terminal state.
The diagram shows the logic flow from request initiation to completion or ID return.
Sources: stacklet/mcp/platform/graphql.py:157-190, stacklet/mcp/platform/tools.py:143-167
Core Components
Configuration Models
The system relies on Pydantic models to structure export requests and parse results.
| Model | Purpose | Key Fields |
|---|---|---|
ExportColumn | Defines a single CSV column mapping. | name, path, subpath |
ExportParam | Defines filters for the GraphQL connection. | name, type, value |
ConnectionExport | Represents the status and result of an export job. | dataset_id, success, download_url, processed_rows |
Sources: stacklet/mcp/platform/models.py:72-132
JMESPath Transformations
Users can apply JMESPath expressions to JSON fields during export. This allows for data manipulation, such as calculating the length of an array or extracting specific values from complex objects.
# Example: Extracting a specific tag value
{"name": "env_tag", "path": "data", "subpath": "Tags[?Key=='Environment'].Value | [0]"}
Sources: stacklet/mcp/platform/dataset_info.md:25-33
Execution and Monitoring
Export Tool Parameters
The platform_dataset_export tool accepts several parameters to control the scope and behavior of the data extraction.
| Parameter | Type | Description |
|---|---|---|
connection_field | str | The GraphQL field to export (e.g., accounts, resources). |
columns | list[ExportColumn] | List of objects defining column headers and data paths. |
node_id | str | Optional ID to start the export from a specific node. |
params | list[ExportParam] | Optional parameters for filtering. |
timeout | int | Wait time in seconds. Use 0 for immediate return. |
Sources: stacklet/mcp/platform/tools.py:143-167
Status Polling Logic
The lookup mechanism uses an exponential backoff strategy when a timeout is specified. The wait_for_export function doubles the interval between checks, starting at 2 seconds, until the timeout is reached or the job completes.
The diagram illustrates the asynchronous polling mechanism between the MCP server and the Platform API.
Sources: stacklet/mcp/platform/graphql.py:175-188, tests/test_tools_platform.py:440-465
Operational Constraints
Filtering and Scope
- Scale Limits: Users should never export all resources blindly. It is recommended to check the total count via
platform_graphql_queryfirst. - Paging Parameters: Connection paging parameters like
firstoraftercannot be used; the export process handles pagination internally. - Expiry: Generated download URLs point to S3 and expire after 24 hours.
Sources: stacklet/mcp/platform/dataset_info.md:5-15, 60-70
Security and Mutations
Dataset exports are generally read-only operations regarding governance data, but they initiate server-side jobs and file creation. As such, they are marked as non-idempotent and non-read-only in tool annotations.
Sources: stacklet/mcp/platform/tools.py:27-35, tests/test_tools_platform.py:490-500
Conclusion
Platform Dataset Exports provide a scalable solution for extracting governance data that exceeds typical API response sizes. By leveraging server-side CSV generation and JMESPath for field-level transformation, the system enables deep analysis of cloud estates while maintaining efficiency through asynchronous job management and status tracking.