Process Management
Relevant source files
The following files were used as context for generating this wiki page:
Process management in Percona Monitoring and Management (PMM) is a multi-layered orchestration system designed to ensure the continuous operation and lifecycle management of both server-side infrastructure and client-side monitoring agents. The system utilizes industry-standard supervisors to manage service state, configuration, and recovery across distributed environments.
The architecture separates concerns between the PMM Server, which manages core backend services like PostgreSQL, ClickHouse, and pmm-managed, and the PMM Client, which employs a dedicated supervisor component to orchestrate exporters and monitoring agents on targeted nodes.
Sources: CONTRIBUTING.md:25-50, README.md:40-45
Server-Side Orchestration
On the PMM Server, process management is primarily handled by supervisord. This component is responsible for bootstrapping the server environment and maintaining the lifecycle of internal database engines and application logic.
Supervisor Configuration and Initialization
The PMM Server configuration is dynamically rendered through templates and stored in pmm.ini. This configuration defines the priorities, start/stop signals, and logging redirects for all core PMM components.
The diagram shows the priority-based startup sequence managed by supervisord within the PMM Server.
Sources: managed/services/supervisord/pmm_config.go:88-150
Core Managed Services
The following services are defined within the supervisord lifecycle:
| Program | Priority | Command | Purpose |
|---|---|---|---|
pmm-init | -1 | ansible-playbook | Performs initial environment setup and provisioning. |
postgresql | 1 | postgres | Internal storage for configuration, models, and metadata. |
clickhouse | 2 | clickhouse-server | High-performance storage for query analytics data. |
nginx | 4 | nginx | Reverse proxy and web server for the UI and API. |
pmm-managed | 14 | pmm-managed | Core backend managing server components and API. |
pmm-agent | 15 | pmm-agent | Client-side agent running locally on the server node. |
Sources: managed/services/supervisord/pmm_config.go:88-165, managed/CONTRIBUTING.md:120-130
Client-Side Process Management
On monitored nodes, pmm-agent acts as the primary process supervisor. It manages a variety of "Agents" (typically exporters) through a specialized Supervisor component that responds to state changes requested by the PMM Server.
Agent Supervisor Lifecycle
The Supervisor tracks the state of child processes using an internal state machine. It handles the transition of processes through various statuses, ensuring that configurations (like listen ports and environment variables) are correctly applied before execution.
This flowchart illustrates the logic used by the agent supervisor to synchronize local process states with the server's desired state.
Sources: agent/agents/supervisor/supervisor_test.go:48-120, agent/agents/supervisor/supervisor_test.go:215-230
Template-Based Configuration
Processes are often started with dynamic parameters. The supervisor uses a template rendering system to inject environment variables, paths, and ports into configuration files or CLI arguments.
- Listen Ports: Automatically assigned from a configured range (e.g., 65000-65099).
- TextFiles: Dynamic configuration files (like
my.cnfor certificates) are rendered to a temporary directory before the process starts. - Environment Variables: Credentials and connection strings are injected via environment variables using
.server_username,.server_password, and other template keys.
Sources: agent/agents/supervisor/supervisor_test.go:240-280, agent/agents/supervisor/supervisor_test.go:420-450
External Orchestration with Nomad
PMM utilizes HashiCorp Nomad as a flexible cluster scheduler to orchestrate client components, particularly in Kubernetes or multi-node container environments. When Nomad is used, the PMM Supervisor manages the Nomad agent process itself, which in turn manages individual monitoring tasks.
Nomad Integration Workflow
pmm-managedsends a Nomad configuration template topmm-agent.pmm-agentrenders the template, injecting local paths for data directories and TLS certificates.- The Supervisor starts the
nomad agentwith araw_execdriver to execute monitoring binaries.
Sources: agent/agents/supervisor/supervisor_test.go:300-400, CONTRIBUTING.md:48-52
Logging and Diagnostics
Process management includes robust log redirection to facilitate troubleshooting. Both supervisord and the agent Supervisor capture stdout and stderr from child processes.
- Supervisord Logs: Located in
/srv/logs/, includingpmm-managed.log,postgresql14.log, andclickhouse-server.log. - Log Management: Logs are typically rotated by the supervisor (e.g.,
stdout_logfile_maxbytes = 50MB). - ClickHouse Specifics: Internal ClickHouse logging to files is often disabled in
config.xmlin favor of supervisord redirection to ensure a unified logging stream.
Sources: managed/services/supervisord/pmm_config.go:140-165, dev/clickhouse-config.xml:45-55
Summary
Process management in PMM provides a resilient framework for orchestrating complex database monitoring workflows. By leveraging supervisord for stable server-side infrastructure and a custom Go-based supervisor in pmm-agent for dynamic client-side exporters, the system ensures that monitoring remains active even during configuration changes or service failures. The integration of template rendering and external schedulers like Nomad further allows PMM to scale across diverse infrastructure types from bare metal to Kubernetes.