High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
PMM Dump is a specialized utility designed to facilitate the transfer of metrics and Query Analytics (QAN) data between Percona Monitoring and Management (PMM) Server instances. It serves primarily as a troubleshooting tool for Percona Services engineers, allowing them to export data from a source PMM server into a compressed archive and subsequently import that data into a destination PMM server.
The tool supports three primary data streams: core metrics stored in VictoriaMetrics, QAN data stored in ClickHouse, and internal PMM metadata stored in PostgreSQL. The architecture is built around a "Transferer" component that orchestrates multiple workers to handle concurrent chunk-based data processing.
Sources: README.md:1-5, cmd/pmm-dump/main.go:61-75
System Components
The architecture consists of the CLI entry point, a central transfer engine, and source-specific implementations for various PMM databases.
Core Architecture Overview
The diagram above illustrates how the Transferer coordinates data flow between the various PMM database sources and the local dump file via a concurrent worker pool.
Sources: cmd/pmm-dump/main.go:210-245, README.md:189-193
Component Descriptions
| Component | Responsibility | Relevant Files |
|---|---|---|
| Transferer | Orchestrates the export/import logic, manages the worker pool, and handles the tar.gz file stream. | pkg/transferer/ |
| VictoriaMetrics Source | Handles time-series metrics via /api/v1/export and /api/v1/import endpoints. Supports native and JSON formats. | pkg/victoriametrics/source.go |
| ClickHouse Source | Handles QAN data using TSV format. Allows filtering via WHERE statements. | pkg/clickhouse/ |
| Postgres Source | Manages internal metadata export/import for grafana, ssmDB, and percona databases. | cmd/pmm-dump/util.go |
| Load Checker | Monitors PMM instance load (CPU/RAM) to postpone or stop exports based on thresholds. | cmd/pmm-dump/main.go |
Sources: cmd/pmm-dump/main.go:61-105, cmd/pmm-dump/util.go:377-387, pkg/victoriametrics/source.go:34-45
Data Export Logic
The export process involves calculating time ranges, splitting those ranges into manageable chunks, and streaming them into a compressed archive.
Export Workflow
- Initialization: The tool validates PMM version support (minimum 2.12.0) and configures connection URLs for the various databases.
- Chunking: Time ranges are split into chunks. VictoriaMetrics uses
chunk-time-range(default 5m), while ClickHouse useschunk-rows. - Concurrency: A pool of workers (configurable via
--workers) pulls chunk metadata from a pool and executes the read from the source. - Load Monitoring: During export, the
LoadCheckerevaluates thresholds for CPU and RAM. If thresholds are exceeded, the export is postponed or halted.
Sources: cmd/pmm-dump/main.go:230-265, cmd/pmm-dump/util.go:47-49, pkg/victoriametrics/source.go:340-358
This sequence shows the interaction between the Transferer, the LoadChecker for safety, and the source specific implementations during an export operation.
Sources: cmd/pmm-dump/main.go:270-302, pkg/victoriametrics/source.go:60-100
Data Import Logic
The import process reads the meta.json from the dump and pushes chunks back to the destination PMM server.
- Format Detection: The tool reads
meta.jsonto determine if VictoriaMetrics data is innativeorjsonformat. - Content Limiting: During import to VictoriaMetrics, the tool can split large chunks if the destination server's Nginx configuration limits request sizes (configurable via
--vm-content-limit). - Finalization: After VictoriaMetrics import, the tool calls
/internal/resetRollupResultCacheto ensure the new data is immediately visible.
Sources: cmd/pmm-dump/main.go:345-380, pkg/victoriametrics/source.go:240-275, pkg/victoriametrics/source.go:319-338
Configuration and Connectivity
Connectivity is managed through environment variables or CLI flags. The tool provides fallbacks to automatically derive component URLs from the primary PMM URL.
| Parameter | Env Var | Fallback Behavior |
|---|---|---|
| VictoriaMetrics URL | PMM_VM_URL | PMM_URL + /prometheus |
| ClickHouse URL | PMM_CLICKHOUSE_URL | PMM_URL (host) + :9000 |
| PostgreSQL URL | PMM_POSTGRES_URL | Derived from PMM_POSTGRES_ADDR or PMM defaults |
Sources: pkg/util/util.go:33-55, pkg/util/util.go:106-118
Redaction and Security
The tool implements URL redaction for logging and metadata storage to prevent credential leakage. The redactURL function uses a regular expression to replace user:pass strings with ***:***.
Sources: cmd/pmm-dump/util.go:420-430, cmd/pmm-dump/security_test.go:94-106
Dump File Structure
The output is a Gzip-compressed TAR archive containing specific directories for each data source.
The dump file organization ensures that different data types are isolated, allowing the tool to selectively import only core metrics or QAN data as requested.
Sources: README.md:189-195, internal/test/e2e/validate_test.go:305-325
Summary
The PMM Dump architecture provides a robust, concurrent mechanism for data portability between PMM instances. By decoupling the transfer logic from the specific database implementations (VictoriaMetrics, ClickHouse, Postgres), the tool maintains extensibility while ensuring data integrity through structured metadata and safe load-aware export procedures.
Sources: cmd/pmm-dump/main.go:240-250, README.md:1-10