The Transferer Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Transferer Engine is the core orchestration component of the pmm-dump utility. Its primary purpose is to facilitate the high-performance movement of metrics and Query Analytics (QAN) data between Percona Monitoring and Management (PMM) instances. It abstracts the complexities of interacting with multiple data backends, such as VictoriaMetrics for core metrics and ClickHouse for QAN data, providing a unified interface for both export and import operations.
The engine utilizes a worker-pool pattern to process data in parallel chunks, ensuring efficiency while maintaining safety through integrated load-checking mechanisms. This allows the tool to pause or terminate operations if the source PMM server experiences excessive CPU or RAM pressure during the export process.
Architecture and Core Components
The Transferer Engine is built around the Transferer struct, which manages the lifecycle of data movement. It coordinates between data sources, the filesystem (or STDOUT/STDIN), and concurrency workers.
Data Structures
| Structure | Description | Source |
|---|---|---|
Transferer | The main engine struct containing registered sources, worker count, and the file handle. | pkg/transferer/transferer.go:33-37 |
Chunk | Represents a single unit of data, including its content, metadata, and source type. | pkg/transferer/import.go:103-110 |
ChunkPool | An interface providing the next chunk metadata to be processed during export. | pkg/transferer/transferer.go:53-55 |
LoadStatusGetter | An interface used to monitor the health of the PMM server during operations. | pkg/transferer/transferer.go:57-59 |
The following diagram illustrates the relationship between the Transferer and its dependencies:
The diagram shows how the Transferer engine orchestrates multiple data sources and monitors system load via defined interfaces.
Sources: pkg/transferer/transferer.go:33-59
Export Logic and Flow
The export process is designed to be highly concurrent but resource-aware. It splits the requested time range into discrete chunks which are then fetched by multiple workers.
Concurrency Model
The engine initializes a specific number of goroutines (defaulting to the number of CPUs) to read chunks from the configured sources. These chunks are passed through a channel (chunksCh) with a limited buffer size (maxChunksInMem = 4) to a single writer goroutine that serializes the data into a compressed Gzip Tar archive.
Load Management
During export, the engine frequently checks the LoadStatusGetter. It supports three states:
- LoadStatusOK: Continues processing normally.
- LoadStatusWait: Pauses the reading workers for a predefined duration (
MaxLoadWaitDuration) to let the server recover. - LoadStatusTerminate: Aborts the export to prevent server failure.
Flowchart demonstrating the concurrent export loop with integrated load-threshold checks.
Sources: pkg/transferer/export.go:31-77, pkg/transferer/transferer.go:28
Import Logic and Flow
The import process reverses the export flow, reading a compressed dump and distributing data back to the appropriate PMM backends.
Processing Steps
- Archive Extraction: The engine opens the Gzip stream and iterates through the Tar headers.
- Metadata Verification: It looks for
meta.jsonto compare PMM and tool versions between the export and import environments. - Chunk Distribution: Data chunks are identified by their directory prefix (e.g.,
vm/for VictoriaMetrics,ch/for ClickHouse). - Parallel Writing: Workers pick up chunks from a channel and call the
WriteChunkmethod of the corresponding source implementation.
Sequence diagram showing the flow of data from a dump file to the destination data sources during import.
Sources: pkg/transferer/import.go:33-96
Supported Data Sources
The Transferer Engine supports multiple backends through the dump.Source interface. Each source is responsible for its own chunking logic and API communication.
| Source Type | Backend | Data Type | Implementation Details |
|---|---|---|---|
VictoriaMetrics | VictoriaMetrics | Core Time-series | Supports native and JSON formats; uses time-range chunking. |
ClickHouse | ClickHouse | QAN Data | Uses row-count based chunking; data transferred via TSV. |
PostgreSQL | PostgreSQL | Configuration/Dashboards | Includes Grafana and PMM internal databases. |
Sources: pkg/clickhouse/source.go:121-168, pkg/victoriametrics/source.go:58-94, cmd/pmm-dump/main.go:210-235
Source Selection Logic
The engine dynamically determines which sources to activate based on CLI flags such as --dump-core, --dump-qan, and --dump-postgres. During import, it matches files found in the archive to the sources provided at initialization.
Sources: cmd/pmm-dump/main.go:61-65, pkg/transferer/import.go:138-155
Metadata Management
The engine handles a meta.json file inside every dump. This file contains critical information for ensuring data integrity:
- PMM Server Version: To detect potential incompatibilities between different PMM releases.
- Arguments: The CLI flags used to generate the dump.
- VMDataFormat: Indicates if VictoriaMetrics data is in
nativeorjsonformat.
Sources: pkg/transferer/meta.go:101-127, cmd/pmm-dump/util.go:183-228
Conclusion
The Transferer Engine provides a robust framework for PMM data portability. By combining a concurrent worker model with strict load monitoring and a modular source architecture, it ensures that even large datasets can be moved safely and efficiently without compromising the performance of the monitoring infrastructure.