Troubleshooting & Error Handling

Troubleshooting and Error Handling in pmm-dump are centered around robust validation of environment compatibility, managing server resource constraints, and providing detailed diagnostic logs. The tool is designed to prevent data loss or server instability by checking PMM Server versions and monitoring resource thresholds during high-volume data transfers.

The system employs a combination of pre-flight checks, concurrent worker management, and specialized error handling for large data payloads to ensure reliability when moving metrics between Percona Monitoring and Management (PMM) instances. Sources: README.md, cmd/pmm-dump/main.go

Compatibility and Version Validation

Before initiating any data transfer, pmm-dump validates the compatibility between the tool and the target PMM Server. The tool requires a minimum PMM Server version of 2.12.0 to function correctly.

Version Check Logic

The validation process queries the PMM API endpoint /v1/server/version. If the server returns a version lower than the minimum requirement, the tool terminates with a fatal error to prevent unpredictable behavior.

Resource Load Management

To prevent pmm-dump from overloading a production PMM Server, it features a Load Checker that monitors CPU and RAM thresholds. This is critical during the export command.

Threshold Configuration

Users can define max-load and critical-load values. If the PMM Server exceeds these percentages, the tool will either postpone the export (waiting for load to drop) or stop the process entirely.

• CPU: Overall CPU load of the PMM instance.
• RAM: RAM load of the PMM instance.
• MYRAM: RAM load of the specific instance running the pmm-dump binary.

Sources: README.md, cmd/pmm-dump/main.go:88-92

Load Monitoring Flow

Handling Large Data Payloads (413 Errors)

A common issue during the import process is the "413 Request Entity Too Large" error, typically caused by Nginx or Apache limits on the PMM Server. pmm-dump provides a specific mechanism to handle this.

Content Limiting Architecture

The --vm-content-limit flag allows users to restrict the size of individual data chunks sent to VictoriaMetrics. When this limit is exceeded, the tool implements a recursive splitting strategy.

1. Detection: The tool identifies a 413 error from the VictoriaMetrics /api/v1/import endpoint.
2. Splitting: If the chunk is too large, pmm-dump splits the metrics within that chunk into smaller sub-chunks.
3. Recursive Retry: It attempts to send these sub-chunks, splitting them again if they still exceed the server's client_max_body_size.

Sources: pkg/victoriametrics/source.go:167-208, internal/test/e2e/contentlimit_test.go:65-75

Implementation Detail: Recursive Splitting

func (s Source) splitMetrics(metricChunks [][]Metric, limit uint64) ([][]Metric, error) {
    // ... logic to divide metrics in half ...
    for _, chunk := range newMetricChunks {
        compressedData, err := compressChunk(chunk)
        if len(compressedData) > int(limit) {
            return s.splitMetrics(newMetricChunks, limit) // Recursive split
        }
    }
    return newMetricChunks, nil
}

Sources: pkg/victoriametrics/source.go:185-208

Diagnostic Logging and Debugging

pmm-dump uses the zerolog library for structured logging. Troubleshooting often requires enabling verbose output to trace worker activity and HTTP requests.

Debug Mode

Enabling --verbose or -v increases the log level to Debug and adds caller information and Goroutine IDs to each log entry. This is essential for identifying which concurrent worker encountered an issue.

• Standard Logs: Time-formatted console output (Stderr).
• Verbose Logs: Includes goroutine_id and file/line references.
• Internal Logs: During export, logs are also written to a dump.log file inside the .tar.gz archive for post-mortem analysis.

Sources: cmd/pmm-dump/main.go:110-128, cmd/pmm-dump/util.go:39-51, README.md

Troubleshooting E2E Tests

For developers contributing to the project, the E2E testing framework includes specific troubleshooting features:

• Persistent Deployments: If an E2E test fails, the Docker test environment is kept alive (not deleted) to allow manual inspection of the PMM Server state.
• Temp Dump Inspection: Some tests store temporary dump files in ./test/pmm/tmp, which can be extracted to verify the integrity of the meta.json or data chunks (vm/ and ch/ directories).

Sources: internal/test/README.md, README.md

Summary

Troubleshooting pmm-dump involves verifying the PMM Server version (>= 2.12.0), adjusting load thresholds (--max-load) to prevent server saturation, and configuring --vm-content-limit to bypass proxy body size restrictions. For complex failures, enabling --verbose mode provides the necessary granularity to trace errors across concurrent export/import workers.