Exporting Data

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/pmm-dump/main.go
• pkg/transferer/export.go
• README.md
• pkg/victoriametrics/source.go
• pkg/postgres/source.go
• cmd/pmm-dump/util.go

The Exporting Data feature in pmm-dump is designed to extract metrics and Query Analytics (QAN) data from a Percona Monitoring and Management (PMM) Server. Its primary purpose is to facilitate the transfer of troubleshooting data to Percona engineers by creating a compressed archive containing time-series data, database query logs, and essential metadata.

The export process is highly configurable, allowing users to define specific time ranges, filter by service instances or dashboards, and manage server load through threshold-based throttling.
Sources: README.md:15-18, cmd/pmm-dump/main.go:81-83

Core Architecture and Data Flow

The export system utilizes a producer-consumer model where multiple worker goroutines read data "chunks" from various sources (VictoriaMetrics, ClickHouse, PostgreSQL) and a single writer goroutine serializes these chunks into a .tar.gz archive.

This diagram illustrates the progression from user input to the final archive, highlighting the concurrent reading and centralized writing.
Sources: pkg/transferer/export.go:35-73, cmd/pmm-dump/main.go:215-245

Concurrent Execution

The Transferer.Export function manages the lifecycle of the export. It initializes a chunksCh channel with a maximum capacity of maxChunksInMem to manage memory usage while reading.

• Read Workers: Multiple goroutines (defined by --workers) execute readChunksFromSource, pulling work from a ChunkPool.
• Write Worker: A single goroutine executes writeChunksToFile, ensuring thread-safe sequential writing to the compressed tar archive.
  Sources: pkg/transferer/export.go:44-67

Load Throttling and Safety

To prevent PMM Server degradation during intensive exports, the system implements a LoadChecker. It monitors CPU and RAM metrics against user-defined thresholds.

Threshold Level	Action	Default Values
Max Load	Postpones reading by sleeping for MaxLoadWaitDuration.	CPU=70, RAM=80, MYRAM=10
Critical Load	Terminates the export process to protect the server.	CPU=90, RAM=90, MYRAM=30

Sources: cmd/pmm-dump/main.go:101-104, pkg/transferer/export.go:88-100

The sequence above details the interaction between workers and the load monitoring system during the data retrieval phase.
Sources: pkg/transferer/export.go:88-108

Data Sources and Chunking

Data is split into logical chunks to facilitate concurrent processing and avoid massive memory allocations.

VictoriaMetrics (Core Metrics)

VictoriaMetrics data is exported via the /api/v1/export or /api/v1/export/native endpoints. The time range is divided into segments based on the --chunk-time-range parameter (default: 5 minutes).
Sources: pkg/victoriametrics/source.go:61-75, pkg/victoriametrics/source.go:289-307

ClickHouse (QAN Data)

Query Analytics data is retrieved from ClickHouse. Chunks are determined by the number of rows, controlled by the --chunk-rows flag (default: 100,000).
Sources: cmd/pmm-dump/main.go:98, cmd/pmm-dump/main.go:255-257

PostgreSQL

If enabled via --dump-postgres, the tool uses pg_dump to export internal PMM databases such as grafana, ssmDB, and percona.
Sources: pkg/postgres/source.go:41-53, pkg/postgres/source.go:27-29

Configuration Options

The export command supports several flags to filter and control the output:

Flag	Type	Description
--start-ts	String	Start RFC3339 date-time for metrics.
--end-ts	String	End RFC3339 date-time for metrics.
--ts-selector	String	Time series selector passed to VictoriaMetrics.
--instance	Strings	Service names to filter both QAN and core metrics.
--dashboard	Strings	Grafana dashboard names to derive metric selectors.
--stdout	Boolean	Redirects archive output to STDOUT for piping.
--vm-native-data	Boolean	Use VM native format (reduces size, may break compatibility).

Sources: cmd/pmm-dump/main.go:85-112, README.md:73-81

Output Structure

The resulting dump is a tar.gz file containing a specific directory hierarchy:

• meta.json: JSON object containing PMM version, tool arguments, and timezone info.
• vm/: Directory containing VictoriaMetrics data chunks (.bin files).
• ch/: Directory containing ClickHouse data chunks in TSV format.
• dump.log: Captured logs from the export execution.

Sources: README.md:142-148, pkg/transferer/export.go:146-150

Implementation Example: Chunk Splitting

The following snippet demonstrates how VictoriaMetrics time ranges are sliced into manageable segments for the workers:

func SplitTimeRangeIntoChunks(start, end time.Time, delta time.Duration) []dump.ChunkMeta {
	var chunks []dump.ChunkMeta
	chunkStart := start
	for {
		s, e := chunkStart, chunkStart.Add(delta)
		chunks = append(chunks, dump.ChunkMeta{
			Source: dump.VictoriaMetrics,
			Start:  &s,
			End:    &e,
		})

		chunkStart = e
		if chunkStart.After(end) {
			break
		}
	}
	return chunks
}

Sources: pkg/victoriametrics/source.go:289-307

The export process concludes by writing the meta.json file and the dump.log into the archive, providing a complete package for data portability.
Sources: pkg/transferer/export.go:136-143