Real-time Analytics
Relevant source files
The following files were used as context for generating this wiki page:
Real-time Analytics within Percona Monitoring and Management (PMM) is a specialized system designed to capture, aggregate, and visualize database query performance metrics as they occur. It focuses on identifying slow queries, error rates, and resource utilization across multiple database engines, including MySQL, MongoDB, and PostgreSQL. This functionality is primarily surfaced through the Query Analytics (QAN) interface, providing users with a "single pane of glass" for troubleshooting and optimizing database performance.
The system relies on a distributed architecture where agents (e.g., rta-mongodb-agent) collect telemetry from monitored instances and push it to a central collector service. This data is then processed and stored in ClickHouse for high-performance analytical querying, allowing the PMM UI to render detailed reports and sparklines that reflect current database load and query behavior.
Sources: README.md, managed/services/telemetry/config.default.yml:710-752, api/qan/v1/collector.proto:9-13
Architecture and Data Flow
The Real-time Analytics pipeline starts at the database instance level and ends in the PMM Dashboards. Data is collected in buckets and transmitted via gRPC to the PMM Server.
Data Collection and Aggregation
Metrics are gathered by various agents such as pmm-agent, vmagent, and specialized RTA agents like rta-mongodb-agent. These agents aggregate query data into MetricsBucket messages to reduce network overhead. A MetricsBucket contains a query's fingerprint (SQL signature without values), execution time statistics, and metadata such as service name, database name, and environment labels.
Sources: api/qan/v1/collector.proto:16-45, managed/services/telemetry/config.default.yml:710-726
Collector Service
The CollectorService exposes an internal rpc Collect that accepts CollectRequest payloads containing multiple MetricsBucket objects. This service is responsible for receiving the data from pmm-agent and preparing it for storage.
Sources: api/qan/v1/collector.proto:9-14
Data Flow Diagram
The diagram shows the path of query metrics from the database through collection agents to the analytical storage and finally the user interface.
Sources: api/qan/v1/collector.proto, managed/services/supervisord/supervisord.go:374-386
Query Analytics (QAN) Metrics
QAN tracks a comprehensive set of metrics categorized by database engine. These metrics are used to calculate "Load" (average active queries) and "QPS" (queries per second).
Core Metrics (Cross-Engine)
| Metric | Description | Units |
|---|---|---|
load | Average Active Queries calculated as query_time / time_range | load |
num_queries | Total count of queries executed in the bucket | QPS |
query_time | Average statement execution time | load (time) |
Sources: api/qan/v1/qan.proto:12-29, dashboards/pmm-app/src/pmm-qan/panel/QueryAnalytics.constants.ts:687-738
Database Specific Metrics
The system provides deep visibility into engine-specific internals:
- MySQL: Tracks InnoDB I/O operations, row locks (
innodb_rec_lock_wait), temporary table creation (tmp_disk_tables), and filesort operations. - MongoDB: Monitors document scans (
docs_examined), response lengths, and lock acquisition counts (global, database, and collection levels). - PostgreSQL: Captures block cache hits (
shared_blks_hit), WAL (Write-ahead logging) records, and CPU user/system time.
Sources: api/qan/v1/qan.proto:34-118, dashboards/pmm-app/src/pmm-qan/panel/QueryAnalytics.constants.ts:77-686
Reporting and API
The GetReport API endpoint is the primary method for the frontend to retrieve aggregated analytics.
Report Request Structure
Users can filter and group analytics by various dimensions using the GetReportBody.
The class diagram illustrates the parameters available for generating a query analytics report, allowing for flexible grouping and filtering.
Sources: api/qan/v1/json/client/qan_service/get_report_responses.go:174-206
Report Response Data
The response contains Rows, where each row represents a dimension (e.g., a specific query fingerprint) and its associated metrics and sparklines.
- Sparklines: A series of
Pointobjects representing metric changes over time within the requested interval. - Metrics Map: A dictionary of metrics where each entry contains statistics like
avg,sum,max,min, andp99.
Sources: api/qan/v1/json/client/qan_service/get_report_responses.go:476-525, api/qan/v1/qan.proto:14-118
System Configuration and Telemetry
Real-time Analytics behavior is managed via supervisord and monitored through internal telemetry.
Configuration via supervisord
The qan-api2 service is managed by supervisord with specific environment variables for database connectivity and data retention.
- Retention: Managed via
--data-retentionflag, defined in days. - Connectivity: Uses
PMM_CLICKHOUSE_ADDR,PMM_CLICKHOUSE_USER, andPMM_CLICKHOUSE_PASSWORD.
Sources: managed/services/supervisord/supervisord.go:374-398
Internal Telemetry
PMM tracks the health and usage of the Real-time Analytics system itself, including:
- Agent Versions: Distribution of
rta-mongodb-agentversions. - Session Durations: How long RTA agents remain active.
- Collection Intervals: The frequency at which agents push data (typically converted to seconds).
Sources: managed/services/telemetry/config.default.yml:710-752
Real-time Analytics provides the necessary infrastructure to transform raw database activity into actionable performance insights. By leveraging bucketed collection, high-performance analytical storage, and engine-specific metric tracking, it enables PMM to provide detailed query-level troubleshooting in real-time.