Checker & Reaper Mechanics
Relevant source files
The following files were used as context for generating this wiki page:
The Checker and Reaper mechanics form the core operational lifecycle of Kuberhealthy. The system is designed to provision synthetic tests (Checks) and one-off tasks (Jobs) as Kubernetes pods, monitor their execution, and ensure the automated cleanup (Reaping) of historical resources. This architectural pattern ensures that the cluster remains clean while providing continuous verification of application and cluster health.
Sources: README.md:90-106, cmd/kuberhealthy/kuberhealthy.go:167-184
Checker Lifecycle and Execution
Kuberhealthy operates as an operator that manages the execution of KuberhealthyCheck (khcheck) and KuberhealthyJob (khjob) resources. The execution logic is primarily handled by the Checker struct in the external package, which manages the transition of a check from creation to reporting.
Execution Flow
When a check is triggered, Kuberhealthy generates a unique UUID for the run and updates the corresponding khstate resource to whitelist only that specific run for reporting. It then provisions a pod based on the PodSpec defined in the check resource.
External Check Reporting
Checker pods communicate their results back to Kuberhealthy via the /externalCheckStatus endpoint. The master Kuberhealthy instance validates the reporting pod using its source IP or a specific kh-run-uuid header to ensure only the whitelisted pod can influence the check state.
Sources: cmd/kuberhealthy/kuberhealthy.go:870-910, pkg/checks/external/main.go:50-55
Reaper Mechanics
The Reaper is a background process that runs on the master Kuberhealthy pod. Its primary responsibility is to delete old checker pods, khjob resources, and orphaned khstate resources to prevent resource exhaustion and clutter in the Kubernetes API.
Pod Reaping Logic
The Reaper monitors pods with the label kuberhealthy-check-name. It evaluates pods based on their completion time and their exit status (Succeeded or Failed).
| Configuration Option | Default Value | Description |
|---|---|---|
MaxCheckPodAge | 30s (Min) | Maximum age of a completed pod before it is reaped. |
MaxCompletedPodCount | 4 | Number of successful pods to keep per check name. |
MaxErrorPodCount | 4 | Number of failed pods to keep per check name. |
minKHJobAge | 5m | Minimum age for a khjob resource before deletion. |
Sources: cmd/kuberhealthy/reaper.go:18-35, deploy/helm/kuberhealthy/values.yaml:8-13
Reaper Operational Flow
The reaper runs on a configurable interval (defaulting to 30 seconds). It executes two main functions: runCheckReap for pod cleanup and runJobReap for khjob custom resource cleanup.
State and Resource Synchronization
Kuberhealthy maintains an internal "State Reflector" to cache the current status of all checks. This system ensures that the status page and metrics endpoint remain performant even with a high volume of checks.
khState Resource Reaper
In addition to pod and job reaping, Kuberhealthy includes a khStateResourceReaper. This specific logic audits khstate resources and deletes any that no longer have a corresponding khcheck or khjob definition. This prevents "ghost" statuses from appearing on the status page for deleted tests.
Sources: cmd/kuberhealthy/kuberhealthy.go:275-305, cmd/kuberhealthy/kuberhealthy.go:310-360
Summary
The Checker mechanics enable a secure, decentralized testing environment where Kuberhealthy acts as an orchestrator for synthetic workloads. Complementing this, the Reaper ensures the system remains self-cleaning by managing the retention of pods and custom resources based on age and quantity thresholds. Together, these systems provide a robust framework for continuous cluster verification without manual resource management.
Sources: README.md:90-110, cmd/kuberhealthy/reaper.go:210-230