Performance & Load Management
Relevant source files
The following files were used as context for generating this wiki page:
Performance and load management in pmm-dump are critical to ensure that exporting or importing large volumes of metrics data does not destabilize the Percona Monitoring and Management (PMM) server. The tool provides mechanisms to throttle operations based on the PMM instance's resource utilization and to manage data transfer throughput via concurrent workers and chunking strategies.
The system utilizes a LoadChecker to monitor thresholds for CPU and RAM usage on either the PMM server or the local instance running the tool. Furthermore, it manages network and memory pressure during imports through content size limits and chunk splitting.
Sources: README.md:129-131, cmd/pmm-dump/main.go:275-279
Load Thresholds and Monitoring
The export process can be automatically postponed or terminated based on pre-defined load thresholds. These thresholds monitor the resource usage of the PMM instance to prevent impact on its primary monitoring duties.
Threshold Categories
The tool supports three primary metric types for load management:
- CPU: Overall CPU load of the PMM instance (0-100%).
- RAM: RAM load of the PMM instance (0-100%).
- MYRAM: RAM load of the local instance executing
pmm-dump(0-100%).
Sources: README.md:144-148, cmd/pmm-dump/main.go:94-98
Load Management Logic
When executing an export, the tool parses max-load and critical-load flags. The LoadChecker component uses these values to determine the state of the transfer. If max-load is exceeded, the export is postponed; if critical-load is exceeded, the export is stopped to protect the server.
The diagram shows the decision flow during a chunk export based on load validation.
Sources: cmd/pmm-dump/main.go:275-281, README.md:129-138
Throughput Control
To optimize performance, pmm-dump employs concurrent workers and data chunking.
Worker Management
The number of concurrent reading/writing workers can be adjusted using the --workers flag. This allows users to scale the transfer speed based on available network bandwidth and server capacity.
Sources: cmd/pmm-dump/main.go:58, README.md:52
Chunking Strategies
Data is split into manageable segments called "chunks" to prevent long-running queries and high memory consumption.
- Core Metrics (VictoriaMetrics): Split by time ranges using
chunk-time-range(defaulting to 5 minutes). - QAN Metrics (ClickHouse): Split by row count using
chunk-rows(defaulting to 100,000 rows).
Sources: cmd/pmm-dump/main.go:88-92, pkg/victoriametrics/source.go:264-282
Import Performance and Content Limits
During data import, especially to VictoriaMetrics, large chunks can lead to 413 Request Entity Too Large errors if they exceed the PMM server's (or its reverse proxy's) intake limits.
VictoriaMetrics Content Splitting
The --vm-content-limit flag enables an automatic splitting mechanism. If a chunk's content exceeds the specified byte limit, the victoriametrics.Source recursively splits the metrics until each fragment fits within the limit.
The sequence diagram illustrates how the VMSource handles chunks exceeding the content limit by fragmenting them before transmission.
Sources: pkg/victoriametrics/source.go:174-205, internal/test/e2e/contentlimit_test.go:49-80
Configuration Options Summary
| Flag | Default | Description |
|---|---|---|
workers | System defined | Number of concurrent workers for processing. |
max-load | CPU=70,RAM=80,MYRAM=10 | Threshold to postpone export operations. |
critical-load | CPU=90,RAM=90,MYRAM=30 | Threshold to stop export operations immediately. |
vm-content-limit | 0 (Disabled) | Maximum size in bytes for VictoriaMetrics import requests. |
chunk-time-range | 5m | Duration of each VictoriaMetrics data chunk. |
chunk-rows | 100000 | Number of rows per ClickHouse data chunk. |
Sources: cmd/pmm-dump/main.go:88-103, README.md:41-65
Conclusion
Performance and Load Management in pmm-dump is designed to be a "safe-by-default" system. By combining active resource monitoring (load thresholds) with controllable concurrency (workers) and data fragmentation (chunking and content limits), the tool provides the necessary tunables to perform massive data migrations without impacting the availability of the PMM environment.
Sources: README.md:129-131, pkg/victoriametrics/source.go:193-195