Prometheus Metrics & Exporter
Relevant source files
The following files were used as context for generating this wiki page:
Kuberhealthy includes a built-in Prometheus exporter that translates the internal state of synthetic checks into Prometheus-compatible metrics. This system allows operators to monitor the health of Kubernetes clusters and individual applications by scraping the /metrics endpoint. The exporter provides granular details on check statuses, run durations, and cluster-wide health states.
Sources: README.md:28-30, pkg/metrics/exporter.go:42-55
Metric Generation Logic
The metrics exporter transforms the project's internal health.State into a string format following Prometheus standards. It supports two primary categories of metrics: system-level metrics (e.g., whether Kuberhealthy itself is running) and check/job-level metrics (status and duration for specific synthetic tests).
Core Components and Data Structures
The exporter relies on the PromMetricsConfig structure to control how labels are rendered, specifically allowing for the suppression or truncation of error messages in the metric labels.
Sources: pkg/metrics/exporter.go:17-20
Metric Formatting
The promMetricName helper function constructs the metric string with appropriate labels. Labels include the check name, namespace, status (0 or 1), and optionally an error string where pipes (|) are used as separators for multiple errors.
Sources: pkg/metrics/exporter.go:23-40
Exported Metrics Reference
Kuberhealthy exposes Gauges that represent binary states (healthy/unhealthy) and durations in seconds.
| Metric Name | Type | Description |
|---|---|---|
kuberhealthy_running | Gauge | 1 if Kuberhealthy is running error-free, 0 otherwise. Includes current_master label. |
kuberhealthy_cluster_state | Gauge | Global health status of the cluster (1 = OK, 0 = Unhealthy). |
kuberhealthy_check | Gauge | Status of a specific Kuberhealthy check (1 = OK, 0 = Unhealthy). |
kuberhealthy_check_duration_seconds | Gauge | Duration of the most recent check run in seconds. |
kuberhealthy_job | Gauge | Status of a specific Kuberhealthy job. |
kuberhealthy_job_duration_seconds | Gauge | Duration of the most recent job run in seconds. |
Sources: pkg/metrics/exporter.go:42-126
Integration with Prometheus Operator
The project provides native support for the Prometheus Operator through ServiceMonitor and PrometheusRule custom resources. This allows for automated discovery of the exporter and predefined alerting logic.
Data Flow for Scraping
The following diagram illustrates how Prometheus interacts with the Kuberhealthy service to collect metrics.
Predefined Alerting Rules
The prometheusrule.yaml template defines two critical alerts:
- KuberhealthyError: Fires if
kuberhealthy_running < 1for 5 minutes. - ClusterUnhealthy: Fires if
kuberhealthy_cluster_state < 1for 5 minutes.
Sources: deploy/helm/kuberhealthy/templates/prometheusrule.yaml:24-55
Configuration via Helm
The Prometheus integration is highly configurable through Helm values. This includes enabling the resources, setting scrape intervals, and applying custom labels for metric relabeling.
| Value | Default | Description |
|---|---|---|
prometheus.enabled | false | Global toggle for Prometheus resources. |
prometheus.serviceMonitor.enabled | false | Enables creation of the ServiceMonitor resource. |
prometheus.serviceMonitor.endpoints.interval | 15s | How often Prometheus should scrape metrics. |
prometheus.prometheusRule.enabled | false | Enables default alerting rules. |
prometheus.grafanaDashboard.enabled | false | Deploys a ConfigMap containing a Grafana dashboard. |
Sources: deploy/helm/kuberhealthy/values.yaml:18-65
Conclusion
The Prometheus Metrics & Exporter system in Kuberhealthy provides a robust bridge between synthetic check results and cluster monitoring infrastructure. By offering standard Prometheus gauges, automated discovery via ServiceMonitors, and predefined alerting rules, it ensures that failures in either the cluster components or the Kuberhealthy operator itself are visible to administrators.