Development Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• cmd/pmm-dump/main.go
• cmd/pmm-dump/util.go
• internal/test/README.md
• internal/test/deployment/pmm.go
• pkg/victoriametrics/source.go
• pkg/clickhouse/source.go

This guide provides technical documentation for developers working on the pmm-dump tool. It covers the build process, local environment setup, architecture of data sources, and the testing framework.

pmm-dump is designed to transfer metrics and Query Analytics (QAN) data between Percona Monitoring and Management (PMM) Server instances. It utilizes a chunk-based export/import strategy to handle large volumes of data from VictoriaMetrics and ClickHouse.
Sources: README.md:1-5, cmd/pmm-dump/main.go:207-228

Getting Started

Prerequisites and Build

To develop for pmm-dump, you must have Go 1.21+ and Docker installed. The project uses a Makefile to streamline common tasks. The binary is built using the command make build.
Sources: README.md:7-11, README.md:144-147

Local Environment Setup

A local development environment can be instantiated using Docker Compose, providing a PMM Server, PMM Client, and MongoDB instance.
Sources: README.md:144-162

Rule	Description
make up	Sets up docker containers for local development
make down	Shuts down docker containers
make re	Shortcut for down then up
make mongo-reg	Registers MongoDB in PMM
make mongo-insert	Executes MongoDB insert to generate test data

Sources: README.md:154-159

---

Core Architecture and Data Flow

The tool operates by abstracting data locations into "Sources" (VictoriaMetrics, ClickHouse, Postgres) and processing data in "Chunks". The Transferer manages the execution flow between these sources and the dump file (a gzipped tar archive).
Sources: cmd/pmm-dump/main.go:240-252, README.md:135-141

Export Logic Flow

The export process involves calculating timeframes, preparing sources with specific selectors/filters, and orchestrating workers to write data chunks to the archive.

The export command defaults to the last 4 hours if start-ts and end-ts are not provided.
Sources: cmd/pmm-dump/main.go:136-168, cmd/pmm-dump/main.go:261-267

Data Sources Implementation

Sources must implement the dump.Source interface, providing methods to read and write chunks.

• VictoriaMetrics Source: Handles core metrics. It can export in JSON or native VM format. It supports splitting large chunks if they exceed the vm-content-limit.
  Sources: pkg/victoriametrics/source.go:61-100, pkg/victoriametrics/source.go:158-185
• ClickHouse Source: Handles QAN data. It uses a WHERE clause for filtering and splits data into chunks based on a specific row count (chunk-rows).
  Sources: pkg/clickhouse/source.go:104-121, pkg/clickhouse/source.go:214-250

---

Load Management

To prevent overloading the PMM server during export, the tool includes a LoadChecker. It monitors CPU and RAM thresholds and can postpone or stop the export process.
Sources: README.md:118-133, cmd/pmm-dump/main.go:285-293

Default thresholds are set to CPU=70, RAM=80, MYRAM=10.
Sources: cmd/pmm-dump/main.go:88-91

---

Testing Framework

The project maintains a robust suite of end-to-end (e2e) and unit tests.

End-to-End (e2e) Tests

E2E tests use the deployment package to programmatically manage Docker containers. The deployment.Controller creates and destroys PMM environments for each test case.
Sources: internal/test/README.md:46-64, internal/test/deployment/pmm.go:154-178

Test Execution Sequence:

1. Remove existing test deployments.
2. Pull required Docker images.
3. Deploy PMM (Server, Client, Mongo).
4. Run pmm-dump binary against the deployment.
5. Validate results (e.g., chunk comparison).
6. Cleanup (unless a test fails, in which case deployments are kept for troubleshooting).
   Sources: internal/test/README.md:37-43, internal/test/e2e/validate_test.go:44-70

Writing a New E2E Test

Developers should use the //go:build e2e tag. A typical test utilizes deployment.NewController(t) and the util.Binary struct to run the compiled tool.
Sources: internal/test/README.md:46-60

// Example e2e test structure
func TestExample(t *testing.T) {
    ctx := context.Background()
    c := deployment.NewController(t)
    pmm := c.NewPMM("test-name", ".env.test")
    pmm.Deploy(ctx) // Deploys PMM Server & Client via Docker

    b := new(util.Binary)
    // Run export command
    stdout, stderr, err := b.Run("export", "--pmm-url", pmm.PMMURL(), "-d", "dump.tar.gz")
}

Sources: internal/test/README.md:65-85

---

Utility Functions

Technical helpers located in cmd/pmm-dump/util.go manage PMM metadata and environment detection.

• PMM Version Validation: Ensures PMM Server is at least version 2.12.0.
  Sources: cmd/pmm-dump/util.go:37, cmd/pmm-dump/util.go:277-293
• Metadata Composition: The composeMeta function gathers Git commit info, PMM version, and the exact arguments used to run the tool, storing them in meta.json inside the archive.
  Sources: cmd/pmm-dump/util.go:197-251
• URL Redaction: Automatically redacts credentials in log output and metadata.
  Sources: cmd/pmm-dump/util.go:413-421

Conclusion

Development of pmm-dump requires a focus on efficient data streaming and robust error handling across distributed sources. The provided testing infrastructure ensures that changes to source implementations (VictoriaMetrics/ClickHouse) do not result in data loss during the export/import lifecycle.
Sources: internal/test/e2e/validate_test.go:94-135