Alerting System
Relevant source files
The following files were used as context for generating this wiki page:
The Alerting System in PMM provides a unified mechanism for monitoring database performance and environment health through customizable templates and rules. It allows users to define conditions for firing alerts, manage notification channels (Contact Points), and apply policies for alert routing and silencing. The system integrates with Grafana for rule execution and uses VictoriaMetrics (via vmalert) for processing metric-based triggers.
Sources: README.md, ui/apps/pmm/src/contexts/tour/steps/alerting.messages.ts:1-60
Architecture and Components
The system is built on a service-oriented architecture where the managed service orchestrates alerting logic, template management, and synchronization with underlying components like Grafana and VictoriaMetrics.
Core Components
| Component | Description |
|---|---|
| Alerting Service | Responsible for template collection, rule validation, and bridging between the API and Grafana. |
| vmalert | Executes PromQL queries against the metrics database to evaluate alert conditions. |
| Grafana | Provides the user interface for managing alert rules, contact points, and notification policies. |
| Template Store | Stores built-in, SaaS-provided, and user-defined alert templates in YAML format. |
Sources: managed/services/alerting/service.go:50-80, managed/services/supervisord/supervisord.go:420-450
Data Flow for Rule Evaluation
The following diagram illustrates how an alert moves from a metrics query to a notification:
The diagram shows the lifecycle of an alert from metric evaluation by vmalert to delivery via Grafana's notification engine, managed by the PMM Alerting Service.
Sources: managed/services/supervisord/supervisord.go:422-435, ui/apps/pmm/src/contexts/tour/steps/alerting.messages.ts:20-50
Alert Templates
Alert templates are the fundamental building blocks of the alerting system. They define PromQL expressions with placeholders that can be customized when creating specific alert rules.
Template Sources
The system collects templates from four primary sources:
- Built-in: Shipped with PMM releases (path:
/usr/local/percona/alerting-templates). - SaaS: Downloaded from external Percona services.
- User File: Loaded from YAML files in
/srv/alerting/templates. - User API: Created and updated via the PMM API and stored in the database.
Sources: managed/services/alerting/service.go:55-58, api/alerting/v1/alerting.proto:55-65
YAML Structure
Templates use a specific YAML format including name, summary, expression (expr), parameters, duration (for), and severity.
templates:
- name: example_threshold_alert
summary: High CPU Usage
expr: [[ .threshold ]] < cpu_usage_percent
params:
- name: threshold
summary: CPU threshold percentage
unit: '%'
type: float
for: 5m
severity: critical
Sources: managed/services/alerting/service.go:120-150, api-tests/alerting/alerting_test.go:535-560
API Service Methods
The AlertingService provides RPC methods for managing the lifecycle of templates and rules.
Service Operations
| Method | Description | Endpoint |
|---|---|---|
ListTemplates | Retrieves all available alerting templates with support for pagination. | GET /v1/alerting/templates |
CreateTemplate | Adds a new user-defined template via YAML. | POST /v1/alerting/templates |
UpdateTemplate | Modifies an existing User API template. | PUT /v1/alerting/templates/{name} |
DeleteTemplate | Removes a User API template. | DELETE /v1/alerting/templates/{name} |
CreateRule | Creates a concrete alert rule in Grafana based on a template. | POST /v1/alerting/rules |
Sources: api/alerting/v1/alerting.proto:170-205, api/alerting/v1/json/v1.json:20-400
Template and Rule Validation
The service enforces strict validation before creating or updating templates:
- Parameter Mapping: Ensures every placeholder in the
expr(e.g.,[[ .param ]]) has a corresponding definition in theparamssection. - Type Checking: Validates that values provided match defined types (
float,bool,string). - Reserved Prefixes: User templates cannot start with
pmm_orsaas_, which are reserved for internal templates.
Sources: managed/services/alerting/service.go:210-245, managed/services/alerting/service_test.go:80-120
Notification and Management
Alerting extends beyond rule definition into notification routing and suppression.
Contact Points and Policies
- Contact Points: Define integration with third-party tools (ChatOps) for receiving notifications.
- Notification Policies: Determine where, when, and how alerts are routed based on labels.
- Silences: Allow users to temporarily suppress notifications for specific alerting rules during maintenance windows.
Sources: ui/apps/pmm/src/contexts/tour/steps/alerting.messages.ts:27-52
Monitoring vmalert via Supervisord
The PMM Server uses Supervisord to manage the vmalert process. Its configuration includes datasource URLs and rules file paths:
[program:vmalert]
command = /usr/sbin/vmalert
--datasource.url={{ .VMURL }}
--remoteRead.url={{ .VMURL }}
--rule=/srv/prometheus/rules/*.yml
--httpListenAddr={{ .InterfaceToBind }}:8880
Sources: managed/services/supervisord/supervisord.go:420-440
Summary
The Alerting System integrates metrics evaluation, template-based rule generation, and flexible notification routing. By separating the logic of alert templates from the specific configuration of rules, the system enables both standardized monitoring (via built-in Percona templates) and highly specific user-defined triggers. The reliance on Grafana for delivery and VictoriaMetrics for evaluation ensures high performance and compatibility with standard observability tools.