Operator Architecture
Relevant source files
The following files were used as context for generating this wiki page:
Kuberhealthy is a Kubernetes operator designed for synthetic monitoring and continuous process verification. It operates by replicating real-world workflows within a cluster to verify that applications and the underlying Kubernetes infrastructure are functioning as expected. By using Custom Resource Definitions (CRDs), the operator manages the lifecycle of synthetic tests, schedules "checker pods," and aggregates health data for external monitoring systems like Prometheus.
Sources: README.md:5-10, deploy/helm/kuberhealthy/README.md:1-5
Core Components and Workflow
The Kuberhealthy architecture centers on the main operator pod which observes custom resources and orchestrates the execution of verification logic. The operator acts as a control plane for "checks" (recurrent tests) and "jobs" (one-off or specific external tasks).
Synthetic Testing Workflow
The operator follows a specific sequence to provision and operate checker pods:
- Observation: Kuberhealthy observes the creation or existence of a
KuberhealthyCheck(khcheck) resource. - Scheduling: It schedules a "checker pod" specifically for that check.
- Execution: The checker pod interacts with the Kubernetes API to deploy resources (e.g., a DaemonSet) to verify cluster functionality.
- Cleanup: After verifying the state (e.g., waiting for pods to be Ready), the checker pod deletes the test resources.
- Reporting: The checker pod reports results back to the Kuberhealthy API.
- Aggregation: Kuberhealthy stores the state and serves it via a JSON status page and Prometheus metrics endpoint.
Sources: README.md:105-125, deploy/helm/kuberhealthy/README.md:5-10
The diagram above illustrates the relationship between the Kubernetes API, the internal logic of the KH Operator, and the checker pods it manages.
Sources: README.md:105-125
Custom Resource Definitions (CRDs)
Kuberhealthy defines two primary custom resources to manage its monitoring tasks: KuberhealthyCheck and KuberhealthyJob.
KuberhealthyCheck (khcheck)
Used for recurrent synthetic tests. These resources define the container image to run, the run interval, and the timeout duration for the test.
| Field | Type | Description |
|---|---|---|
runInterval | string | How often the check should be executed (e.g., "15m"). |
timeout | string | Maximum time allowed for the check to complete. |
podSpec | object | Standard Kubernetes PodSpec defining the checker container. |
Sources: deploy/helm/kuberhealthy/crds/comcast.github.io_khchecks.yaml:45-60
KuberhealthyJob (khjob)
Used for configuring external checker jobs. Unlike checks, jobs focus on specific phases and completion rather than strict intervals.
Sources: deploy/helm/kuberhealthy/crds/comcast.github.io_khjobs.yaml:35-50
External Metrics and Integration
Kuberhealthy exposes health data through multiple channels to facilitate integration with existing observability stacks.
- Status Page: Served at port 80 on the
kuberhealthy.kuberhealthyservice. It provides a globalOKboolean and granularCheckDetails. - Prometheus Metrics: Available at the
/metricsendpoint. The operator converts check results into Prometheus-compatible metrics automatically. - InfluxDB: Supports metric forwarding to InfluxDB instances.
Sources: README.md:25-35, README.md:150-160
This sequence diagram shows how the checker pod reports its findings and how the operator subsequently provides that data to Prometheus.
Sources: README.md:150-170
High Availability and Failover
The operator supports deployment with multiple replicas (defaulting to 2 in Helm configurations). However, replicas primarily act in a failover capacity rather than a load-balancing capacity for the operator's management logic. The current "Master" pod is identified in the JSON status output to indicate which instance is authoritatively managing the checks.
Sources: deploy/helm/kuberhealthy/README.md:65-75, README.md:195-200
Conclusion
The architecture of Kuberhealthy leverages the Kubernetes Operator pattern to automate the testing of cluster health. By decoupling the test execution (Checker Pods) from the reporting logic (Operator API), it provides a scalable and language-agnostic platform for synthetic monitoring. Its reliance on standard Kubernetes primitives (CRDs and PodSpecs) ensures deep integration with cluster security and scheduling policies.