Dump File Format
Relevant source files
The following files were used as context for generating this wiki page:
The PMM Dump file is a structured container designed to facilitate the transfer of metrics and Query Analytics (QAN) data between Percona Monitoring and Management (PMM) Server instances. It serves as a portable, compressed archive that encapsulates data from various sources including VictoriaMetrics, ClickHouse, and PostgreSQL, along with critical metadata and execution logs.
Physically, the dump is a tar archive compressed using gzip (standard .tar.gz format). Its internal structure is organized into directories representing different data sources, ensuring that core metrics, query data, and service information remain segmented and identifiable during the import process.
Sources: README.md:104-106, pkg/transferer/export.go:121-127
Archive Structure
The archive uses a standardized directory layout to separate different types of exported data. Each directory corresponds to a specific SourceType defined in the codebase.
| Path | Description | Data Format |
|---|---|---|
meta.json | Global metadata about the dump and PMM environment. | JSON |
log.json | Execution logs captured during the export process. | JSON Lines / Text |
vm/ | Core metrics data chunks from VictoriaMetrics. | Native VM or JSON |
ch/ | Query Analytics (QAN) data chunks from ClickHouse. | TSV |
pg/ | PostgreSQL database dumps (e.g., Grafana, ssmDB). | Custom pg_dump |
Sources: README.md:108-112, pkg/dump/dump.go:21-22, pkg/dump/source.go:37-45
Visualizing the Archive Layout
The following diagram illustrates the relationship between the compressed archive and its internal file components.
The diagram shows the hierarchical structure of the exported .tar.gz file.
Sources: README.md:108-112, pkg/transferer/export.go:153-157
Metadata Specification
The meta.json file is a critical component located at the root of the archive. It contains environment details from the source PMM server and information about how the export was executed.
Meta Data Model
The Meta struct defines the schema for this file:
| Field | Type | Description |
|---|---|---|
version | PMMDumpVersion | Contains Git branch and commit of the tool used. |
pmm-server-version | string | The version of the source PMM Server. |
max_chunk_size | int64 | The size of the largest data chunk in bytes. |
pmm-server-timezone | *string | Preferred timezone configured in the source Grafana. |
arguments | string | Redacted CLI arguments used for the export. |
vm-data-format | string | Format of VM data (native or json). |
pmm-server-services | []PMMServerService | List of monitored services and their IDs. |
Sources: pkg/dump/dump.go:24-33, pkg/transferer/meta.go:81-89
Service Information
When the --export-services-info flag is used, the pmm-server-services field is populated with detailed inventory data including service names, node IDs, node names, and agent IDs.
Sources: pkg/dump/dump.go:35-40, cmd/pmm-dump/main.go:94
Data Chunks
Data is not exported as a single monolithic file but is instead split into "chunks" to manage memory usage and allow for parallel processing.
Chunk Types and Naming
Each chunk is stored within its respective source directory. The filenames are typically generated based on the chunk's metadata, such as time ranges or indices.
- VictoriaMetrics (vm): Chunks are named using the start and end Unix timestamps (e.g.,
START-END.bin). They can be in VictoriaMetrics' native format or a JSON-based format. - ClickHouse (ch): Chunks represent QAN data rows, typically exported in TSV format.
- PostgreSQL (pg): Dumps created via
pg_dumpwith the custom format option, named asINDEX.pgdump.
Sources: pkg/dump/dump.go:42-53, pkg/victoriametrics/source.go:91-95, pkg/postgres/source.go:56-58
Processing Flow
The Transferer coordinates the creation of these chunks through a pool mechanism.
This diagram shows the flow of data from the source component into the final archive structure.
Sources: pkg/transferer/export.go:121-140, pkg/dump/dump.go:76-92
Data Format Compatibility
The dump format supports two primary modes for VictoriaMetrics data, configurable via the --vm-native-data flag:
- JSON Format: Standard format that is highly compatible across different PMM versions but results in larger dump sizes.
- Native Format: VictoriaMetrics' internal export format. It significantly reduces dump size but may have compatibility limitations between different PMM versions.
The format used is recorded in the vm-data-format field of meta.json to ensure the importer uses the correct API endpoints (/api/v1/import vs /api/v1/import/native).
Sources: README.md:109, cmd/pmm-dump/main.go:315-333, pkg/victoriametrics/source.go:193-197
Conclusion
The PMM Dump file format provides a robust and version-aware container for PMM data. By utilizing standard archive formats (tar.gz) and structured metadata (meta.json), it ensures that telemetry data can be reliably transported and validated across disparate environments. The use of source-specific subdirectories and chunk-based storage allows the tool to handle large datasets while maintaining granular control over the import/export process.