Introduction to PMM Dump

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• cmd/pmm-dump/main.go
• cmd/pmm-dump/util.go
• pkg/dump/dump.go
• pkg/victoriametrics/source.go
• pkg/postgres/source.go
• pkg/util/util.go

PMM Dump is a technical tool designed to facilitate the transfer of metrics and Query Analytics (QAN) data between Percona Monitoring and Management (PMM) Server instances. Its primary purpose is to assist Percona Services engineers in troubleshooting issues by allowing them to export specific timeframes of data from a source PMM server and import them into a target environment for analysis.

The tool operates by interacting with the underlying data stores of PMM, including VictoriaMetrics for core metrics, ClickHouse for QAN data, and PostgreSQL for management data. It supports both a command-line interface for manual operations and standard I/O piping for use in automated pipelines.

Sources: README.md:1-5, cmd/pmm-dump/main.go:88-110

Core Architecture and Data Flow

PMM Dump is structured as a Go-based CLI application that utilizes a "Source" abstraction to handle different database backends. The application logic is divided between command parsing, environment configuration, and a transferer module that manages the concurrent processing of data chunks.

High-Level Operation Flow

The following diagram illustrates the lifecycle of an export and import operation:

Component Interaction

PMM Dump interacts with multiple PMM components:

• Grafana API: Used for authentication, fetching PMM version, and retrieving service metadata.
• VictoriaMetrics: Source and destination for core time-series metrics.
• ClickHouse: Source and destination for QAN (Query Analytics) data.
• PostgreSQL: Handled via pg_dump and pg_restore for internal PMM data.

Sources: cmd/pmm-dump/util.go:73-200, pkg/util/util.go:34-58

Data Structure and Formats

The Dump File Structure

The output of an export is a tar.gz archive containing organized data chunks and metadata:

File/Directory	Description
meta.json	JSON object containing PMM version, tool arguments, and service info.
vm/	VictoriaMetrics data chunks, split by timeframe.
ch/	ClickHouse data chunks, split by row counts (TSV format).
log.json	Logs generated during the export process.

Sources: README.md:162-167, pkg/dump/dump.go:31-34

Metadata Schema

The meta.json file is critical for ensuring compatibility during import.

type Meta struct {
	Version           PMMDumpVersion     `json:"version"`
	PMMServerVersion  string             `json:"pmm-server-version"`
	MaxChunkSize      int64              `json:"max_chunk_size"`
	PMMTimezone       *string            `json:"pmm-server-timezone"`
	Arguments         string             `json:"arguments"`
	VMDataFormat      string             `json:"vm-data-format"` // "native" or "json"
	PMMServerServices []PMMServerService `json:"pmm-server-services,omitempty"`
}

Sources: pkg/dump/dump.go:36-45

Key Features and Commands

Export and Import Operations

The tool provides granular control over what data is processed through various flags.

Feature	Description	Relevant Flags
Data Filtering	Filter by time, service name, or dashboard.	--start-ts, --end-ts, --instance, --dashboard
Load Protection	Pause or stop export if PMM CPU/RAM exceeds thresholds.	--max-load, --critical-load, --ignore-load
Data Formats	Support for VictoriaMetrics' native format for smaller size.	--vm-native-data
Concurrency	Adjustable worker count for parallel processing.	--workers

Sources: README.md:21-65, cmd/pmm-dump/main.go:50-80

Resource Management (Load Checker)

During export, the tool monitors the PMM server's health to prevent performance degradation. It checks:

1. CPU Load: PMM instance CPU utilization (0-100%).
2. RAM Load: PMM instance RAM utilization.
3. MYRAM: The memory consumption of the pmm-dump process itself.

Sources: README.md:150-160, cmd/pmm-dump/main.go:277-285

Implementation Details: VictoriaMetrics Source

The Source for VictoriaMetrics handles data chunking by splitting the requested timeframe into smaller intervals (default 5 minutes).

For imports, if the target server has restrictive body size limits (e.g., Nginx client_max_body_size), the tool can split chunks further based on the --vm-content-limit flag.

Sources: pkg/victoriametrics/source.go:174-210

Configuration and Environment

PMM Dump prioritizes configuration from command-line flags, followed by environment variables, and finally falls back to deriving URLs from the base PMM_URL.

Variable	Fallback Logic
PMM_CLICKHOUSE_URL	Derived from PMM_URL (host + port 9000).
PMM_VM_URL	Derived from PMM_URL + /prometheus.
PMM_POSTGRES_URL	Manual entry required or defaults to local.

Sources: pkg/util/util.go:34-150, cmd/pmm-dump/util.go:323-370

Conclusion

PMM Dump is a specialized utility that bridges the gap between separate PMM installations. By abstracting the complexities of ClickHouse, VictoriaMetrics, and PostgreSQL, it provides a unified interface for data portability, reinforced by safety features like load monitoring and chunk-based processing.