Pipeline Integration
Relevant source files
The following files were used as context for generating this wiki page:
Pipeline integration in pmm-dump refers to the ability to stream metric and Query Analytics (QAN) data directly between export and import processes using standard I/O streams (STDOUT and STDIN). This feature enables a seamless transfer of data from one Percona Monitoring and Management (PMM) Server instance to another without the necessity of creating intermediate physical dump files on disk.
This functionality is primarily triggered by the --stdout flag during the export command, which redirects the tar.gz archive stream to the standard output. The import command automatically detects when it is receiving data through a pipe (STDIN), allowing for direct consumption of the export stream.
Sources: README.md:104-108, cmd/pmm-dump/main.go:303-306
Architecture and Data Flow
The pipeline architecture relies on the Go io.ReadWriteCloser interface to abstract the data destination (for export) and source (for import). When operating in a pipeline, the application bypasses filesystem operations in favor of os.Stdout and os.Stdin.
Stream Redirection Logic
The system determines whether to use a physical file or a standard stream based on the provided CLI arguments and the state of the terminal device.
The diagram shows how data flows through the system when the export and import commands are linked via a shell pipe.
Sources: cmd/pmm-dump/util.go:375-408, cmd/pmm-dump/main.go:210-214
Implementation Details
The core logic for pipe detection is handled by the checkPiped function, which examines the mode of the standard input.
| Component | Logic Reference | Description |
|---|---|---|
| Detection | checkPiped() | Uses os.Stdin.Stat() and os.ModeCharDevice to determine if input is coming from a pipe. |
| File Creation | createFile() | If piped is true, returns os.Stdout; otherwise, creates a file at dumpPath. |
| File Retrieval | getFile() | If piped is true, returns os.Stdin; otherwise, opens the file at dumpPath. |
Sources: cmd/pmm-dump/util.go:341-351, cmd/pmm-dump/util.go:375-408
Configuration and Limitations
When running in a pipeline, certain metadata and formatting features behave differently because the tool cannot seek through the stream like it can with a file on disk.
Metadata Handling
During a piped import, the tool cannot read the meta.json file contained within the tar.gz archive before starting the processing of data chunks. Consequently, the user must explicitly provide certain format flags that would otherwise be auto-detected.
- VictoriaMetrics Format: If
--vm-native-datais not provided during a piped import, the tool defaults to VictoriaMetrics JSON format. - Version Support: The tool still performs version checks against the PMM API before starting the transfer to ensure compatibility.
Sources: cmd/pmm-dump/main.go:252-271, cmd/pmm-dump/util.go:361-373
Command Line Interface
The primary command for a direct transfer between two PMM servers is:
./pmm-dump export --pmm-url="http://src-pmm" --stdout | ./pmm-dump import --pmm-url="http://dest-pmm"
Supported Parameters for Pipelines
| Parameter | Command | Description |
|---|---|---|
--stdout | export | Redirects the dump archive to STDOUT. |
--vm-native-data | import | Required if the source stream was exported in VictoriaMetrics native format. |
--dump-qan | Both | Must be specified on both sides of the pipe to transfer QAN data. |
--dump-core | Both | Enabled by default; transfers VictoriaMetrics series. |
Sources: README.md:104-108, cmd/pmm-dump/main.go:100-100, cmd/pmm-dump/main.go:273-275
Sequence of Piped Execution
The following sequence diagram illustrates the lifecycle of a piped operation between two instances of pmm-dump.
The sequence demonstrates the continuous flow of data chunks from the exporter to the importer via the system pipe.
Sources: cmd/pmm-dump/main.go:237-300, pkg/victoriametrics/source.go:204-230, pkg/clickhouse/source.go:167-185
Usage in Automated Environments
Pipeline integration is frequently used in containerized environments and support scripts to automate PMM data recovery or migration.
Support Script Integration
The load-pmm-dump support script utilizes the tool's import capabilities to automatically populate volatile PMM server containers. It specifically handles the removal of Nginx body size limits (client_max_body_size) which can otherwise interfere with large piped transfers.
Sources: support-files/README.md:1-25
Error Handling in Pipes
If a piped import fails due to size constraints, the tool suggests using the --vm-content-limit flag. However, this flag is explicitly not supported when using VictoriaMetrics native data format in a pipeline.
Sources: cmd/pmm-dump/main.go:294-300, cmd/pmm-dump/main.go:273-275
Conclusion
Pipeline integration provides a high-efficiency path for data migration between PMM instances. By leveraging standard UNIX pipes, pmm-dump minimizes disk I/O and temporary storage requirements. While it requires more explicit configuration for data formats (such as native vs. JSON), it remains the preferred method for real-time troubleshooting and server-to-server transfers.