Kubernetes Workload Checks
Relevant source files
The following files were used as context for generating this wiki page:
Kubernetes Workload Checks are a suite of synthetic monitors designed to continuously verify the functional health of a Kubernetes cluster. Unlike traditional passive monitoring, these checks perform active operations—such as creating deployments, scheduling cronjobs, or validating resource quotas—to ensure that the cluster's control plane and underlying infrastructure are operating as expected.
These checks are implemented as containerized applications orchestrated by the Kuberhealthy operator. Each check reports its success or failure back to the Kuberhealthy API, which then aggregates the results into a centralized status page and Prometheus metrics endpoint.
Sources: README.md:1-25, deploy/helm/kuberhealthy/values.yaml:205-345
Deployment Check Architecture
The Deployment Check verifies that the cluster can successfully manage the lifecycle of a Deployment and its associated Service. It ensures that pods can be scheduled, pass traffic through a Service endpoint, and (optionally) perform rolling updates without dropping connections.
Operational Logic
The check follows a specific sequence of actions to validate the workload:
- Cleanup: Identifies and removes orphaned services and deployments from previous runs.
- Provisioning: Applies a deployment configuration (defaulting to 2 replicas) and waits for all pods to reach a
Readystate. - Service Validation: Creates a
ClusterIPservice and waits for its endpoint to become active. - Traffic Verification: Performs an HTTP GET request to the service hostname, expecting a
200 OKresponse. - Rolling Update (Optional): If enabled, updates the container image and validates that the service remains reachable during and after the transition.
- Tear Down: Deletes the resources and verifies successful removal.
Sources: cmd/deployment-check/README.md:1-35, cmd/deployment-check/main.go:133-165
Flow of Deployment Verification
This diagram illustrates the lifecycle of a single deployment check run.
CronJob Health Monitoring
The CronJob checker ensures that scheduled tasks are being executed within their defined schedule windows. This is critical for catching issues where the Kubernetes cron-controller fails to schedule jobs despite the CronJob resource appearing healthy.
Validation Mechanism
The checker iterates through CronJobs in a specified namespace and performs the following logic:
- Schedule Parsing: Uses the
cronexprlibrary to determine when the CronJob should have last run based on itsspec.schedule. - Window Calculation: Defines a 10-minute window around the predicted run time.
- Status Comparison: Compares the
status.lastScheduleTimefrom the cluster against the calculated window. - Failure Reporting: If the last run time is outside the window or missing (for established jobs), a failure is reported.
Sources: cmd/cronjob-checker/main.go:65-115
CronJob Verification Sequence
The sequence below shows how the checker interacts with the Kubernetes API to validate schedules.
Sources: cmd/cronjob-checker/main.go:50-130
Resource Quota Verification
The Resource Quota check monitors whether namespaces are approaching their allocated limits for CPU and memory. This prevents "silent" scheduling failures where controllers cannot create pods due to quota exhaustion.
Thresholds and Filtering
The check allows for granular control over which namespaces are monitored and what constitutes a "near-failure" state:
- Threshold: A configurable float (default
0.9) representing 90% utilization. - Blacklist/Whitelist: Environment variables
BLACKLISTandWHITELISTfilter the namespaces processed by the checker.
Sources: cmd/resource-quota-check/README.md:5-25
| Variable | Description | Default |
|---|---|---|
THRESHOLD | The percentage of quota used before reporting an error. | 0.9 |
BLACKLIST | Comma-separated list of namespaces to ignore. | default |
WHITELIST | Comma-separated list of namespaces to explicitly include. | kube-system,kuberhealthy |
Sources: cmd/resource-quota-check/README.md:35-45
Configuration and Environment Variables
Workload checks are configured primarily through environment variables passed to the checker pods via the KuberhealthyCheck custom resource.
Common Configuration Options
The following table summarizes the configuration parameters for the Deployment Check.
| Environment Variable | Description | Default Value |
|---|---|---|
CHECK_IMAGE | Initial container image for the deployment. | nginxinc/nginx-unprivileged:1.17.8 |
CHECK_IMAGE_ROLL_TO | Image used for the rolling update test. | nginxinc/nginx-unprivileged:1.17.9 |
CHECK_DEPLOYMENT_REPLICAS | Number of replicas to maintain. | 2 |
CHECK_DEPLOYMENT_ROLLING_UPDATE | Enables the rolling update logic. | false |
CHECK_NAMESPACE | Namespace where test resources are created. | kuberhealthy |
SHUTDOWN_GRACE_PERIOD | Time allowed for cleanup after an interrupt. | 30s |
Sources: cmd/deployment-check/README.md:65-90, cmd/deployment-check/main.go:100-130
Resource Definitions (Example)
A standard KuberhealthyCheck for a deployment workload is defined in YAML as follows:
apiVersion: comcast.github.io/v1
kind: KuberhealthyCheck
metadata:
name: deployment
namespace: kuberhealthy
spec:
runInterval: 10m
timeout: 15m
podSpec:
containers:
- name: deployment
image: kuberhealthy/deployment-check:v1.9.0
env:
- name: CHECK_DEPLOYMENT_REPLICAS
value: "4"
- name: CHECK_DEPLOYMENT_ROLLING_UPDATE
value: "true"
Sources: cmd/deployment-check/README.md:100-120, deploy/helm/kuberhealthy/values.yaml:235-255
Summary
Kubernetes Workload Checks provide a robust mechanism for continuous process verification. By simulating real-world workloads—Deployments, CronJobs, and Resource usage—Kuberhealthy can detect infrastructure regressions that standard health probes might miss, such as CNI failures, Docker daemon errors, or controller-manager bottlenecks. These checks ensure that the cluster is not just "up," but fully capable of performing its core orchestration duties.
Sources: README.md:100-115, cmd/deployment-check/README.md:25-30