VictoriaMetrics Integration

Relevant source files

The following files were used as context for generating this wiki page:

• managed/services/victoriametrics/victoriametrics.go
• managed/services/victoriametrics/scrape_configs.go
• managed/models/victoriametrics_params.go
• managed/services/supervisord/supervisord.go
• managed/services/victoriametrics/victoriametrics_test.go

The VictoriaMetrics integration serves as the primary time-series database engine for Percona Monitoring and Management (PMM). It is responsible for high-performance ingestion, storage, and querying of metrics data collected from various database exporters and PMM components. The integration is managed by the pmm-managed service, which handles configuration generation, validation, and dynamic reloading of VictoriaMetrics components.

Within PMM, VictoriaMetrics operates either as an internal service managed by supervisord or as an external endpoint. The integration scope includes managing scrape configurations for vmagent, handling alerting through vmalert, and providing a Prometheus-compatible API for Grafana dashboards.

Sources: managed/services/victoriametrics/victoriametrics.go:37-60, managed/services/supervisord/supervisord.go:420-445, README.md:33-45

Architecture and Components

The VictoriaMetrics integration consists of several specialized processes and a management service that orchestrates their configuration.

Core Service Components

• VictoriaMetrics Server: The main database engine providing storage and querying capabilities.
• vmagent: A tiny but powerful agent which helps collect metrics from various sources and push them to VictoriaMetrics.
• vmalert: Executes alerting and recording rules against VictoriaMetrics data.
• vmproxy: Proxies requests to VictoriaMetrics, often used for filtering or multi-tenancy support.

Sources: managed/services/supervisord/supervisord.go:420-475, managed/services/victoriametrics/victoriametrics.go:343-370

Component Relationship Diagram

The following diagram illustrates how pmm-managed orchestrates the different VictoriaMetrics components.

The VictoriaMetrics Service in pmm-managed reads inventory data from the database to generate scrape configurations for the runtime processes.
Sources: managed/services/victoriametrics/victoriametrics.go:50-80, managed/services/supervisord/supervisord.go:420-475

Configuration Management

Configuration is dynamic and driven by the state of the PMM inventory. The Service struct in managed/services/victoriametrics is responsible for building the promscrape.yml file used by VictoriaMetrics.

Configuration Lifecycle

The service uses a batching mechanism to prevent excessive reloads when multiple inventory changes occur rapidly. A reloadCh channel with a capacity of 1 is used to signal needed updates, which are then delayed by updateBatchDelay (3 seconds) to allow for batching.

Validation and Safety

Before applying any configuration, pmm-managed performs a "dry run" by executing the victoriametrics binary with the -promscrape.config.dryRun=true flag. If validation fails, the service prevents the configuration from being applied and restores the last known good configuration.

Sources: managed/services/victoriametrics/victoriametrics.go:211-255, managed/services/victoriametrics/victoriametrics.go:257-285

Scrape Configurations

The integration generates complex Prometheus-compatible scrape jobs based on the type of agent and service being monitored.

Job Generation Logic

Scrape configs are divided into several categories:

1. Internal Services: Self-monitoring for victoriametrics, vmalert, grafana, pmm-managed, qan-api2, and clickhouse.
2. External Exporters: Dynamically generated jobs for remote nodes and exporters.
3. High Availability (HA): When HA is enabled, scrape configurations are adjusted based on whether the current node is the leader.

Sources: managed/services/victoriametrics/scrape_configs.go:54-95, managed/services/victoriametrics/victoriametrics.go:303-340

Scrape Configuration Parameters

Parameter	Description	Source
scrape_interval	Frequency of data collection (default 1m global).	MetricsResolutions
scrape_timeout	Timeout for scraping, typically slightly less than interval.	ScrapeTimeout()
honor_timestamps	Whether to use exporter-provided timestamps (default false).	scrape_configs.go:200
metrics_path	The HTTP path to fetch metrics from (e.g., /metrics).	victoriametrics.go:343

Sources: managed/services/victoriametrics/scrape_configs.go:195-215, managed/services/victoriametrics/victoriametrics.go:343-370

External VictoriaMetrics Support

PMM can be configured to use an external VictoriaMetrics instance instead of the bundled one. This is controlled via VictoriaMetricsParams.

• Internal Mode: VMBaseURL points to 127.0.0.1:9090. supervisord is responsible for starting the binary.
• External Mode: Detected if the hostname in the URL is not 127.0.0.1. In this mode, pmm-managed skips health checks and does not attempt to manage the local process via supervisord.

Sources: managed/models/victoriametrics_params.go:29-35, managed/models/victoriametrics_params.go:78-81, managed/services/supervisord/supervisord.go:375-385

Service Management via Supervisord

When running internally, the processes are managed as supervisord programs. The configuration for these programs is rendered from templates.

VictoriaMetrics Program Template

[program:victoriametrics]
command = /usr/sbin/victoriametrics
    --promscrape.config=/etc/victoriametrics-promscrape.yml
    --retentionPeriod={{ .DataRetentionDays }}d
    --storageDataPath=/srv/victoriametrics/data
    --httpListenAddr={{ .InterfaceToBind }}:9090
    --http.pathPrefix=/prometheus

Sources: managed/services/supervisord/supervisord.go:420-440

VMAgent and VMAlert Parameters

Option	Default	Purpose
search.maxQueryLen	1MB	Limits the size of incoming queries.
search.maxQueryDuration	90s	Maximum time a query can run before termination.
promscrape.streamParse	true	Enables memory-efficient parsing of large scrape responses.
retentionPeriod	Based on Settings	How long to keep time-series data.

Sources: managed/services/supervisord/supervisord.go:45-55, managed/services/supervisord/supervisord.go:420-435

Conclusion

The VictoriaMetrics integration provides a robust and scalable telemetry backend for PMM. By leveraging pmm-managed for dynamic configuration and supervisord for process lifecycle management, PMM ensures that the metrics pipeline is automatically updated as the monitored environment changes. The inclusion of vmagent and vmalert extends this capability to support efficient data collection from remote sources and complex alerting logic.