Utility Packages
Relevant source files
The following files were used as context for generating this wiki page:
Utility packages in the pmm-dump project provide essential cross-cutting concerns such as configuration management, HTTP client initialization, logging hooks, and data source preparation. These utilities ensure consistent behavior across the export and import workflows, handling the complexities of PMM (Percona Monitoring and Management) server versions, authentication, and connection strings for backend databases like ClickHouse, VictoriaMetrics, and PostgreSQL.
The scope of these utilities ranges from low-level byte formatting and URL redaction to high-level orchestration of PMM service discovery and metadata composition.
Configuration and Connection Management
The project relies on a centralized configuration structure to manage connections to PMM and its constituent databases. The util package provides logic to resolve these URLs from explicit arguments or environment variables.
PMM Configuration Resolution
The GetPMMConfig function serves as the primary entry point for establishing the connection landscape. It attempts to parse provided links and falls back to environment variables or automatic composition based on the main PMM URL if specific database links are missing.
The diagram above illustrates the fallback logic for resolving database connection strings.
Sources: pkg/util/util.go:34-63
Connection URL Components
| Component | Default Value | Environment Variable |
|---|---|---|
| ClickHouse | clickhouse://default:clickhouse@127.0.0.1:9000/pmm | PMM_CLICKHOUSE_URL |
| VictoriaMetrics | http://127.0.0.1:9090/prometheus | PMM_VM_URL |
| PostgreSQL | postgres://pmm:pmm@127.0.0.1:5432/pmm-managed?sslmode=disable | PMM_POSTGRES_URL |
Sources: pkg/util/util.go:66-136
HTTP and Networking Utilities
The tool utilizes fasthttp for high-performance networking. A specialized helper function, newClientHTTP, configures the client with specific timeouts and TLS settings appropriate for interacting with PMM APIs.
HTTP Client Configuration
The client is restricted to 2 connections per host and implements a 30-second connection wait timeout. It also supports InsecureSkipVerify for environments using self-signed certificates.
Sources: cmd/pmm-dump/util.go:50-64
PMM API Helpers
Utility functions wrap common API calls to the PMM Server to retrieve critical metadata:
- Version Check:
getPMMVersionretrieves the server and full version from/v1/server/version. - Service Discovery:
getPMMServicesiterates through/v1/inventory/services, resolving associated Node IDs and Agent IDs. - Timezone Discovery:
getPMMTimezonefetches Grafana preferences to align data timestamps.
Sources: cmd/pmm-dump/util.go:80-176
Logging and Debugging
A custom logging hook is implemented to assist in debugging concurrent operations by providing visibility into Go runtime internals.
Goroutine Logging
The goroutineLoggingHook extracts the current Goroutine ID using runtime.Stack and attaches it to every log event when verbose mode is enabled. This is crucial for tracing the activity of the workersCount during multi-threaded export/import processes.
Sources: cmd/pmm-dump/util.go:68-78, cmd/pmm-dump/main.go:121-131
Data Transformation and Metadata
Metadata management ensures that dumps are self-describing and compatible across different PMM environments.
Metadata Composition
The composeMeta function aggregates system information into a dump.Meta struct, including:
- Tool version (Git branch and commit).
- PMM Server version and timezone.
- Redacted CLI arguments to prevent credential leakage in the metadata file.
- Service inventory if
export-services-infois enabled.
Sources: cmd/pmm-dump/util.go:185-238
Redaction Logic
To maintain security, the redactURL utility uses regular expressions to strip credentials from connection strings before they are logged or stored in meta.json.
Sources: cmd/pmm-dump/util.go:414-422
Formatters
Two formatting utilities provide human-readable data sizes:
- ByteCountDecimal: Uses base-1000 units (kB, MB).
- ByteCountBinary: Uses base-1024 units (KiB, MiB).
Sources: cmd/pmm-dump/util.go:240-264
Source Preparation Logic
Before data transfer begins, utility functions in main.go and cmd/pmm-dump/util.go prepare the source objects (VictoriaMetrics, ClickHouse, PostgreSQL).
The sequence illustrates the initialization flow for data sources during the command setup phase.
Sources: cmd/pmm-dump/util.go:303-345, cmd/pmm-dump/main.go:167-200
Testing Utilities
The project includes an internal testing deployment utility that automates the lifecycle of PMM environments using Docker.
PMM Deployment Controller
The PMM struct in internal/test/deployment manages container IDs, ports, and network names. It provides helper methods to generate URLs for ClickHouse and MongoDB tailored for the dynamically assigned Docker ports.
Sources: internal/test/deployment/pmm.go:42-127
Summary of Deployment Methods
| Method | Description |
|---|---|
Deploy | Pulls images, creates networks, and starts PMM Server/Client/Mongo containers. |
Restart | Restarts the PMM Server and updates published port mappings. |
Destroy | Stops and removes all containers, volumes, and networks associated with the test. |
PingClickhouse | Ensures the ClickHouse backend is reachable before tests proceed. |
Sources: internal/test/deployment/pmm.go:169-322
Utility packages provide the structural foundation for pmm-dump, abstracting environment-specific details into a set of reliable, reusable functions. By centralizing URL resolution, security redaction, and PMM service discovery, the utility layer allows the core transfer logic to remain focused on data movement.