Custom Resource Definitions (CRDs)
Relevant source files
The following files were used as context for generating this wiki page:
Custom Resource Definitions (CRDs) are the primary mechanism through which Kuberhealthy extends the Kubernetes API to manage synthetic monitoring workloads. They allow users to define desired health checks as native Kubernetes objects, which the Kuberhealthy operator then observes and executes.
The system relies on three core CRDs: KuberhealthyCheck for recurring health tests, KuberhealthyJob for one-off or phased task execution, and KuberhealthyState for persisting the results and status of these workloads.
Sources: README.md:16-25, deploy/helm/kuberhealthy/crds/comcast.github.io_khchecks.yaml:12-25
Core Architecture and Lifecycle
Kuberhealthy functions as a Kubernetes operator that watches for the creation or modification of these custom resources. When a resource is identified, Kuberhealthy schedules a checker pod based on the pod specification provided within the CRD.
The following diagram illustrates the relationship between a user-defined check and the resulting state persistence:
This diagram shows the end-to-end flow from the creation of a CRD configuration to the persistence of health data.
Sources: README.md:95-110, cmd/kuberhealthy/crd.go:17-43
CRD Definitions
KuberhealthyCheck (khcheck)
The KuberhealthyCheck resource defines a recurring synthetic test. It includes configuration for how often the test should run and the containerized logic required to perform the check.
| Field | Type | Description |
|---|---|---|
runInterval | string | Duration string (e.g., "15m") specifying how often to run the check. |
timeout | string | Maximum time allowed for the check to complete. |
podSpec | object | Standard Kubernetes PodSpec defining the container image and environment. |
extraAnnotations | map | Additional annotations to apply to generated checker pods. |
Sources: deploy/helm/kuberhealthy/crds/comcast.github.io_khchecks.yaml:32-60
KuberhealthyJob (khjob)
Similar to checks, KuberhealthyJob represents external checker jobs. Unlike checks, jobs are often used for distinct phases and include a phase field to track their current condition.
The ER diagram shows how jobs interact with state resources to persist their current condition labels.
Sources: deploy/helm/kuberhealthy/crds/comcast.github.io_khjobs.yaml:30-45, cmd/kuberhealthy/crd.go:121-137
KuberhealthyState (khstate)
The KuberhealthyState resource acts as the status storage for both checks and jobs. It is automatically managed by the Kuberhealthy operator and updated when checker pods report their findings.
| Property | Type | Required | Description |
|---|---|---|---|
OK | boolean | Yes | Indicates if the last run was successful. |
Errors | array[string] | Yes | List of error messages if OK is false. |
AuthoritativePod | string | Yes | The specific Kuberhealthy pod that updated this state. |
LastRun | date-time | No | Timestamp of the most recent execution. |
RunDuration | string | Yes | How long the test took to execute. |
khWorkload | string | No | Type of workload: KhCheck or KHJob. |
Sources: deploy/helm/kuberhealthy/crds/comcast.github.io_khstates.yaml:55-100
Controller Interaction and Sanitization
The Kuberhealthy control plane includes specific logic to manage the lifecycle of these CRDs. This includes sanitizing resource names to ensure compatibility with Kubernetes DNS requirements and ensuring that a corresponding khstate resource exists for every check or job.
Name Sanitization
Before interacting with the Kubernetes API for CRD management, names are sanitized. DNS-1123 subdomains must consist of lower case alphanumeric characters, '-' or '.', and start/end with an alphanumeric character.
// sanitizeResourceName cleans up the check names for use in CRDs.
func sanitizeResourceName(c string) string {
// the name we pass to the CRD must be lowercase
nameLower := strings.ToLower(c)
return strings.Replace(nameLower, " ", "-", -1)
}
Sources: cmd/kuberhealthy/crd.go:49-57
State Management Logic
The operator implements several helper functions to maintain state consistency:
- ensureStateResourceExists: Verifies if a state CRD exists for a workload and creates it if missing.
- setCheckStateResource: Updates the
khstateCRD with current status, errors, and an authoritative pod hostname. - setJobPhase: Specifically updates the
phaselabel within aKuberhealthyJobspec.
Sources: cmd/kuberhealthy/crd.go:60-80, cmd/kuberhealthy/crd.go:121-137
Conclusion
Kuberhealthy's CRD-based architecture allows for highly extensible monitoring by treating synthetic tests as first-class Kubernetes citizens. By decoupling the test configuration (khcheck/khjob) from the result persistence (khstate), Kuberhealthy ensures that cluster health information is always queryable via standard kubectl commands and easily exportable to external metrics systems like Prometheus.