Importing Data

Relevant source files

The following files were used as context for generating this wiki page:

• pkg/transferer/import.go
• README.md
• cmd/pmm-dump/main.go
• cmd/pmm-dump/util.go
• pkg/victoriametrics/source.go
• pkg/clickhouse/source.go
• pkg/postgres/source.go
• pkg/dump/source.go

The import process in pmm-dump is designed to ingest metrics and Query Analytics (QAN) data from a compressed dump file into a target PMM Server instance. This feature allows Percona engineers and database administrators to restore or migrate monitoring data across different environments. The system supports three primary data backends: VictoriaMetrics for core metrics, ClickHouse for QAN data, and PostgreSQL for Grafana and internal PMM metadata.

The tool operates by reading a .tar.gz archive, identifying data chunks by their directory structure, and distributing the workload across multiple concurrent workers that communicate with the PMM Server's API endpoints.

Sources: README.md:1-12, pkg/transferer/import.go:34-45

Architectural Overview

The import architecture follows a producer-consumer pattern where the main thread reads the dump file and populates a channel with data chunks, while a pool of worker goroutines consumes these chunks and writes them to the respective database sources.

Core Components

Component	Description
Transferer	Orchestrates the import by opening the dump file and managing worker lifecycles.
Source Interface	Abstraction for database-specific logic (WriteChunk and FinalizeWrites).
Worker Pool	Concurrent goroutines that perform the actual HTTP/DB write operations.
Chunk Channel	A thread-safe queue (chan *dump.Chunk) used to pass data between the reader and workers.

Sources: pkg/transferer/import.go:45-65, pkg/dump/source.go:18-24

Import Execution Flow

The following diagram illustrates the sequence of events during a data import:

The diagram shows the lifecycle of an import operation, from CLI initiation to final backend synchronization.
Sources: pkg/transferer/import.go:34-118, cmd/pmm-dump/main.go:278-333

Data Processing Logic

The import logic handles different file types found within the dump archive. It specifically looks for a meta.json file to perform version checks and then processes directories named after the source types (vm, ch, pg).

Source Type Mapping

The tool identifies the destination backend based on the directory prefix in the tar archive:

• vm/: VictoriaMetrics (Core Metrics)
• ch/: ClickHouse (QAN Data)
• pg/: PostgreSQL (Internal Metadata)

Sources: pkg/dump/source.go:39-50, pkg/transferer/import.go:88-91

Data Chunks Handling

Each chunk is processed individually. If a chunk is empty, it is skipped with a warning. Valid chunks are encapsulated in a dump.Chunk structure containing the source type, filename, and raw content before being sent to the worker channel.

ch := &dump.Chunk{
    ChunkMeta: dump.ChunkMeta{
        Source: st,
    },
    Content:  content,
    Filename: filename,
}

Sources: pkg/transferer/import.go:94-110

Backend Specific Implementations

VictoriaMetrics (Core Metrics)

The VictoriaMetrics source handles the restoration of time-series data. It supports both JSON and Native formats. A critical feature during import is the vm-content-limit, which allows splitting large chunks into smaller ones if the target PMM Server has restrictive client_max_body_size settings in Nginx.

• Endpoint: /api/v1/import or /api/v1/import/native.
• Split Logic: If ContentLimit is set and exceeded, the source recursively splits the metrics list and sends them in multiple requests.

Sources: pkg/victoriametrics/source.go:175-210, cmd/pmm-dump/main.go:330-336

ClickHouse (QAN)

ClickHouse data is expected in TSV format. The import uses prepared SQL statements to insert data efficiently.

• Logic: Reads the TSV records and executes them against the metrics or metrics_local table.
• Finalization: Calls Commit() on the transaction to ensure data persistence.

Sources: pkg/clickhouse/source.go:146-166, pkg/clickhouse/source.go:189-192

PostgreSQL

PostgreSQL restoration leverages the system's pg_restore utility.

• Command: pg_restore --clean --if-exists --dbname <URL>.
• Execution: The chunk content is piped directly into the stdin of the pg_restore process.

Sources: pkg/postgres/source.go:70-89

Configuration and CLI Usage

Importing data requires a target PMM URL and the path to the dump file.

Flag	Envar	Description
--pmm-url	PMM_URL	Full connection string for the target PMM instance.
--dump-path	-	Path to the .tar.gz file created during export.
--workers	-	Number of concurrent import workers (Default: 1).
--vm-content-limit	-	Size limit (bytes) for VictoriaMetrics chunks to prevent 413 errors.

Sources: cmd/pmm-dump/main.go:61-105, README.md:25-50

Piped Import

The tool supports importing data directly from standard input (stdin), enabling pipeline operations without intermediate files:
./pmm-dump export --stdout | ./pmm-dump import --pmm-url="http://target-host"

Sources: cmd/pmm-dump/main.go:297-302, README.md:88-91

Error Handling and Safety

The import process includes several safety checks:

1. Version Support: Checks if the target PMM Server version is at least 2.12.0.
2. Metadata Comparison: Compares the version and configuration in meta.json with the target server to ensure compatibility.
3. Request Size: If a 413 Request Entity Too Large error is encountered from VictoriaMetrics, the tool suggests using the vm-content-limit or decreasing chunk ranges.

Sources: cmd/pmm-dump/util.go:42, cmd/pmm-dump/main.go:330-336, pkg/transferer/import.go:80-84

The import functionality is finalized by calling FinalizeWrites() on all registered sources, ensuring that transactions are committed and caches (like VictoriaMetrics rollup result cache) are reset.
Sources: pkg/transferer/import.go:126-130, pkg/victoriametrics/source.go:242-261