High Availability (HA)
Relevant source files
The following files were used as context for generating this wiki page:
High Availability (HA) in Percona Monitoring and Management (PMM) is a system architecture designed to ensure continuous monitoring service availability by clustering PMM Server nodes. It utilizes a leader-follower model to manage roles and synchronization across the cluster. The system monitors the health and status of nodes through a MemberList-based mechanism, allowing for failover and status reporting within the PMM environment.
Sources: api/ha/v1beta1/ha.swagger.json, managed/services/supervisord/supervisord.go:210-216
Architecture and Cluster Management
The HA architecture relies on multiple nodes operating in specific roles. Nodes within an HA cluster are assigned a role of either LEADER or FOLLOWER. This ensures that there is a central point for certain operations while maintaining redundancy.
The diagram shows the peer-to-peer relationship between nodes in the HA cluster, where roles are defined to manage service coordination.
The system tracks node membership using MemberList, which provides the current operational status of each individual node. This information is exposed through the HAService API, which allows administrators to monitor the cluster's health.
Sources: api/ha/v1beta1/ha.proto:58-64, api/ha/v1beta1/ha.swagger.json
API Service (HAService)
The HAService is the primary interface for interacting with the HA subsystem. It provides endpoints to check the global status of HA and to list the specific details of nodes within the cluster.
Key API Endpoints
| Endpoint | Method | Summary | Description |
|---|---|---|---|
/v1/ha/status | GET | HA Status | Returns whether High Availability mode is "Enabled" or "Disabled". |
/v1/ha/nodes | GET | List HA Nodes | Returns a list of all nodes in the cluster with status and roles. |
Sources: api/ha/v1beta1/ha.swagger.json, api/ha/v1beta1/ha.pb.go:213-231
Node Data Structures
A node in the HA cluster is represented by the HANode message, which contains the following fields:
| Field | Type | Description |
|---|---|---|
node_name | string | Human-readable name of the node. |
role | NodeRole | Role in the cluster (UNSPECIFIED, LEADER, FOLLOWER). |
status | string | Current status derived from MemberList. |
Sources: api/ha/v1beta1/ha.proto:44-50, api/ha/v1beta1/ha.pb.go:78-90
Component Integration and Configuration
High Availability status influences the configuration of various PMM Server components, particularly Grafana and its alerting system. When HA is enabled, specific environment variables are injected into service configurations to enable gossip-based synchronization.
Grafana HA Integration
When HA is enabled, the supervisord service configures Grafana with unified alerting HA parameters. This includes setting the gossip port and advertising the node's address to peers.
Logic flow within the supervisord service for configuring Grafana based on HA status.
Sources: managed/services/supervisord/supervisord.go:210-221, managed/services/supervisord/supervisord.go:343-347
Database and Agent Initialization
During PMM Server setup, if an HA Node ID is provided, the system initializes PMM Server agents in HA mode. This involves:
- Generating unique Agent and Node IDs.
- Configuring
pmm-agentwith the server address and generated ID. - Creating a node record with the
IsPMMServerNodeflag. - Setting up standard exporters (like
node_exporter) attached to the HA-configured PMM Agent.
Sources: managed/models/database.go:885-950
Implementation Details
The HA role enumeration defines the state machine for node priority within the cluster.
Sources: api/ha/v1beta1/ha.proto:58-64
Configuration Parameters
The supervisord service uses a HAParams model to pass cluster-wide settings to configuration templates.
| Parameter | Source/Description |
|---|---|
HAEnabled | Boolean flag indicating if HA is active. |
GrafanaGossipPort | The port used for Grafana alerting synchronization. |
HAAdvertiseAddress | The IP address this node advertises to other cluster members. |
HANodes | A comma-separated list of peer addresses (e.g., node1:9095,node2:9095). |
Sources: managed/services/supervisord/supervisord.go:210-220
Conclusion
High Availability in PMM provides a structured method for maintaining monitoring continuity. By leveraging the HAService API for cluster visibility and integrating with component-level HA features (such as Grafana's unified alerting), PMM ensures that monitoring data and alerts remain available even if individual nodes encounter issues.
Sources: api/ha/v1beta1/ha.proto, managed/services/supervisord/supervisord.go