Introduction to Kuberhealthy
Relevant source files
The following files were used as context for generating this wiki page:
Kuberhealthy is a Kubernetes operator designed for synthetic monitoring and continuous process verification. It functions by running real-world workflows—such as deploying pods, creating daemonsets, or performing DNS lookups—to ensure that a cluster is operating as expected from an application's perspective. By mimicking user behavior rather than just monitoring static resource states, Kuberhealthy identifies complex failures like CNI communication issues, Docker daemon errors, or intermittent API failures that traditional monitoring might miss.
The system is highly extensible, allowing developers to write custom checks in any language. It integrates natively with monitoring ecosystems by providing a Prometheus metrics endpoint (/metrics), a simple JSON status page, and support for InfluxDB metric forwarding. Kuberhealthy is a CNCF-hosted project and serves as a proactive health-checking layer for Kubernetes environments.
Sources: README.md:1-15, deploy/helm/kuberhealthy/README.md:3-10
Architecture and Core Concepts
Kuberhealthy operates as an operator that manages the lifecycle of synthetic tests. It observes Custom Resource Definitions (CRDs) known as KuberhealthyCheck (or khcheck) and khjob to determine which tests to run and how often.
Workflow Lifecycle
The core logic involves Kuberhealthy scheduling "checker pods" based on configured checks. These pods execute specific technical tasks (e.g., creating a deployment) and report the results back to the Kuberhealthy API.
The diagram above shows the lifecycle from check creation to metric exposure.
Sources: README.md:95-108, deploy/helm/kuberhealthy/README.md:15-25
Components and Responsibilities
| Component | Description |
|---|---|
| Kuberhealthy Operator | The central controller that watches for check resources, manages pod scheduling, and aggregates results. |
| Checker Pods | Short-lived pods created to run a specific test (e.g., dns-resolution-check, deployment-check). |
| Kuberhealthy API | An internal HTTP service where checker pods report their status (Success or Failure). |
| Status Page | A JSON endpoint on port 80 showing the global OK status and granular CheckDetails. |
| Metrics Exporter | Converts check results into Prometheus-compatible gauge metrics. |
Sources: README.md:15-30, README.md:144-155, pkg/metrics/exporter.go:48-60
Synthetic Checks
Kuberhealthy comes with a registry of pre-made checks that verify core Kubernetes functionality. These are configured via the Helm values.yaml file or static manifests.
Included Check Examples
- Daemonset Check: Verifies every node in the cluster can provision and tear down pods.
- Deployment Check: Tests rolling updates and traffic passing without dropping connections.
- DNS Check: Validates internal (
kubernetes.default) and external (google.com) resolution. - SSL Handshake: Monitors certificate validity and expiration.
Sources: README.md:118-131, deploy/helm/kuberhealthy/values.yaml:166-250
Creating Custom Checks
Custom checks can be written in any language. The checker pod only needs to hit the Kuberhealthy report endpoint with a JSON payload indicating success or failure.
// Example of a custom Go check using the checkclient package
func main() {
ok := doCheckStuff()
if !ok {
checkclient.ReportFailure([]string{"Test has failed!"})
return
}
checkclient.ReportSuccess()
}
Sources: README.md:135-150, cmd/http-check/main.go:193-201
Monitoring and Observability
Kuberhealthy aggregates the status of all individual checks into a unified view for alerting and visualization.
Prometheus Metrics
The system exports several gauge metrics to indicate cluster health. A value of 1 typically represents "Healthy" or "OK", while 0 represents "Unhealthy".
| Metric Name | Description |
|---|---|
kuberhealthy_running | Indicates if the Kuberhealthy operator itself is running without errors. |
kuberhealthy_cluster_state | Global cluster health status based on all active checks. |
kuberhealthy_check | Status of a specific check (labels include check, namespace, status). |
kuberhealthy_check_duration_seconds | The time taken for a specific check to complete. |
Sources: pkg/metrics/exporter.go:48-115
JSON Status Object
The status page provides a detailed breakdown used for manual inspection or simple automated polling.
{
"OK": true,
"Errors": [],
"CheckDetails": {
"kuberhealthy/daemonset": {
"OK": true,
"RunDuration": "22.51s",
"Namespace": "kuberhealthy",
"LastRun": "2019-11-14T23:24:16Z"
}
}
}
Sources: README.md:162-195
Deployment and Configuration
Kuberhealthy is typically deployed via Helm or static YAML manifests into a dedicated kuberhealthy namespace.
Configuration via Helm
The values.yaml file defines the behavior of the operator, including resource limits and which checks are enabled.
# deploy/helm/kuberhealthy/values.yaml snippet
prometheus:
enabled: true
serviceMonitor:
enabled: true
deployment:
replicas: 2
check:
daemonset:
enabled: true
runInterval: 15m
Sources: deploy/helm/kuberhealthy/values.yaml:1-20, deploy/README.md:20-40
Deployment Logic
Multiple replicas are supported for high availability, but they operate in a failover capacity where only the CurrentMaster manages check scheduling.
The sequence shows how the leader replica coordinates check execution and reporting.
Sources: README.md:95-108, pkg/metrics/exporter.go:66-85
Conclusion
Kuberhealthy provides a specialized monitoring layer for Kubernetes by focusing on the "functional" health of the cluster. By automating complex synthetic tests and exposing them through standard Prometheus metrics, it allows operators to catch "silent" cluster failures before they impact end-users. Its architecture encourages extension through custom checker containers, making it adaptable to specific application requirements.