VictoriaMetrics Integration

Relevant source files

The following files were used as context for generating this wiki page:

• pkg/victoriametrics/source.go
• pkg/victoriametrics/config.go
• pkg/victoriametrics/metric.go
• cmd/pmm-dump/main.go
• cmd/pmm-dump/util.go
• pkg/util/util.go
• pkg/victoriametrics/util.go

The VictoriaMetrics integration in pmm-dump serves as the primary mechanism for exporting and importing "core metrics" from a Percona Monitoring and Management (PMM) server. It facilitates the movement of time-series data between PMM instances by interacting with VictoriaMetrics' export and import APIs.

This module handles the configuration of connection parameters, the partitioning of time-series data into manageable chunks, and the management of data formats (JSON or Native). It also includes logic for handling large payloads through content splitting and ensures cache consistency by resetting rollup caches after write operations.

Sources: README.md, cmd/pmm-dump/main.go:50-80, pkg/victoriametrics/source.go:34-42

Architecture and Components

The integration is centered around the Source struct, which implements the dump.Source interface. It utilizes a Config structure to define how the tool communicates with the VictoriaMetrics instance and what data it targets.

Key Data Structures

• Source: The main handler for VictoriaMetrics operations, containing an HTTP client and configuration.
• Config: Defines connection URLs, time-series selectors, data format preferences, and content limits.
• Metric: Represents the JSON structure of a time-series metric, including labels, values, and timestamps.

Data Export Logic

The export process retrieves data using the VictoriaMetrics /api/v1/export (JSON) or /api/v1/export/native (Native) endpoints.

1. Time Range Partitioning: The total time range is split into discrete chunks based on the chunk-time-range parameter (default 5m).
2. Selector Application: Data is filtered using Time Series Selectors (e.g., {service_name="mongo"}).
3. HTTP Retrieval: Each chunk is requested via GET with Accept-Encoding: gzip.

Data Import and Content Management

The import process sends data to the /api/v1/import or /api/v1/import/native endpoints via POST. A significant feature is the ability to split chunks if they exceed a specific size limit to avoid 413 Request Entity Too Large errors.

Content Splitting Mechanism

When a ContentLimit is set and the data is in JSON format, the Source will:

1. Decompress the chunk.
2. Parse the individual Metric objects.
3. Recursively split the metrics into smaller batches until each batch (when re-compressed) fits within the limit.
4. Send each sub-chunk individually.

Sources: pkg/victoriametrics/source.go:188-232, cmd/pmm-dump/util.go:297-308

Cache Finalization

After all chunks are written, the tool calls /internal/resetRollupResultCache to ensure that VictoriaMetrics serves fresh data based on the newly imported metrics.

Sources: pkg/victoriametrics/source.go:261-275

API Endpoints and Configuration

The following table summarizes the endpoints used by the integration:

Endpoint	Method	Purpose
/api/v1/export	GET	Export metrics in line-delimited JSON format.
/api/v1/export/native	GET	Export metrics in VictoriaMetrics native format.
/api/v1/import	POST	Import metrics in line-delimited JSON format.
/api/v1/import/native	POST	Import metrics in VictoriaMetrics native format.
/internal/resetRollupResultCache	GET	Resets the rollup result cache after import.

Sources: pkg/victoriametrics/source.go:76-80, pkg/victoriametrics/source.go:235-238, pkg/victoriametrics/source.go:262

Configuration Parameters

Parameter	Type	Default	Description
victoria-metrics-url	string	Derived	The base URL for the VictoriaMetrics API.
ts-selector	string	{__name__=~".*"}	Prometheus-style series selectors.
vm-native-data	bool	false	If true, uses native binary format instead of JSON.
vm-content-limit	uint64	0	Limit for POST body size in bytes (JSON format only).
chunk-time-range	duration	5m	The timeframe duration for each exported file.

Sources: cmd/pmm-dump/main.go:68-112, pkg/victoriametrics/config.go:18-23, pkg/util/util.go:87-93

Technical Implementation Details

Version Support and Validation

The tool performs a test request to /api/v1/export during initialization. If a 404 Not Found is returned, it indicates that the PMM server version might not be supported (minimum supported version is 2.12.0).

Sources: pkg/victoriametrics/util.go:28-45, cmd/pmm-dump/util.go:34

Metric Parsing

JSON metrics are parsed from an io.Reader stream using a standard JSON decoder in a loop to handle the line-delimited format efficiently.

Sources: pkg/victoriametrics/metric.go:30-46

// pkg/victoriametrics/metric.go:24-28
type Metric struct {
	Metric     map[string]string `json:"metric"`
	Values     []float64         `json:"values"`
	Timestamps []int64           `json:"timestamps"`
}

Summary

The VictoriaMetrics integration provides a robust pathway for handling large-scale time-series data transfers. By supporting both JSON and Native formats, and implementing intelligent chunking and splitting strategies, it ensures compatibility and performance across different PMM environments. The integration is finalized by a cache reset to maintain data integrity for end-users.