Pod Health Checks

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/pod-restarts-check/main.go
• cmd/pod-status-check/main.go
• cmd/namespace-pod-check/main.go
• pkg/checks/external/main.go
• cmd/kuberhealthy/kuberhealthy.go
• cmd/pod-status-check/README.md
• cmd/pod-restarts-check/README.md

Pod Health Checks in Kuberhealthy are synthetic monitoring tools designed to continuously verify the operational integrity of Kubernetes Pods. These checks replicate real-world workflows—such as deploying, observing, and deleting pods—to identify issues that traditional monitoring might miss, such as nodes stuck in Terminating states, CNI communication failures, or excessive container restarts.

The system operates by deploying specialized "checker pods" that interact with the Kubernetes API to evaluate the state of the cluster. These checkers report their findings (Success or Failure) back to the Kuberhealthy operator via a reporting URL, which then aggregates the data into Prometheus metrics and a JSON status page.
Sources: pkg/checks/external/main.go, cmd/kuberhealthy/kuberhealthy.go, cmd/pod-status-check/README.md

Checker Lifecycle and Architecture

Kuberhealthy manages external checks through a lifecycle that involves provisioning a temporary pod for every check execution. This architecture ensures isolation and allows checks to be written in any language, provided they can communicate with the Kuberhealthy API.

Workflow Execution

The lifecycle of a pod-based health check follows a structured sequence:

1. Trigger: Kuberhealthy observes a KuberhealthyCheck or KuberhealthyJob resource.
2. Provisioning: A checker pod is scheduled with specific environment variables like KH_REPORTING_URL and KH_RUN_UUID.
3. Execution: The checker pod performs its logic (e.g., listing pods, checking events).
4. Reporting: The checker reports results back to the /externalCheckStatus endpoint.
5. Cleanup: Kuberhealthy evicts or deletes the checker pod after completion.

This diagram represents the interaction between the operator, the API server, and the synthetic checker pod.
Sources: pkg/checks/external/main.go:429-570, cmd/kuberhealthy/kuberhealthy.go:945-1080

Specific Health Check Modules

Pod Status Check

The Pod Status Check monitors pods for unhealthy lifecycle phases. It specifically flags pods that remain in Pending, Failed, or Unknown states for longer than a configurable duration (defaulting to 10 minutes).
Sources: cmd/pod-status-check/main.go:88-124, cmd/pod-status-check/README.md

Phase	Health Classification	Description
Running	Healthy	Bound to a node, all containers created.
Succeeded	Healthy	All containers terminated successfully.
Pending	Unhealthy (if > 10m)	Accepted by system but containers not yet created.
Failed	Unhealthy	Containers terminated with non-zero exit code.
Unknown	Unhealthy	State could not be obtained (communication error).
Sources: cmd/pod-status-check/README.md:33-46

Pod Restarts Check

This check monitors for stability by observing BackOff events. It lists Warning events in a target namespace and identifies pods where the restart count exceeds MAX_FAILURES_ALLOWED.
Sources: cmd/pod-restarts-check/main.go:126-155, cmd/pod-restarts-check/README.md

The logic flow for identifying pods caught in crash loops.
Sources: cmd/pod-restarts-check/main.go:128-155

Namespace Pod Check

A comprehensive verification check that ensures pods can be successfully deployed and deleted across all namespaces in the cluster. This validates node-level functionality and admission controller performance across different administrative boundaries.
Sources: cmd/namespace-pod-check/main.go:48-84

Configuration and Environment Variables

External checks are configured via environment variables injected into the checker pods by the Kuberhealthy operator.

Variable	Source File	Purpose
KH_REPORTING_URL	pkg/checks/external/main.go:34	The endpoint where the pod sends JSON results.
KH_RUN_UUID	pkg/checks/external/main.go:38	Unique ID for the current check run to prevent duplicates.
KH_CHECK_RUN_DEADLINE	pkg/checks/external/main.go:41	Unix timestamp for when the check must finish.
SKIP_DURATION	cmd/pod-status-check/main.go:94	Duration to ignore young pods (e.g., "10m").
MAX_FAILURES_ALLOWED	cmd/pod-restarts-check/main.go:73	Threshold for restart events before failing.
Sources: pkg/checks/external/main.go:33-45, cmd/pod-restarts-check/main.go:68-80

Reporting and Result Validation

When a checker pod calls the externalCheckStatus endpoint, Kuberhealthy performs validation to ensure the reporting pod is authorized.

1. UUID Whitelisting: Kuberhealthy maintains a CurrentUUID for every check. If the incoming UUID does not match the whitelisted one, the report is rejected.
2. Identity Verification: The operator identifies the calling pod by its IP or the kh-run-uuid header and cross-references its annotations (e.g., comcast.github.io/check-name) with the active check registry.
   Sources: cmd/kuberhealthy/kuberhealthy.go:975-1030, pkg/checks/external/main.go:403-460

Implementation Example: Reporting Failure

// From cmd/namespace-pod-check/main.go
func ReportFailureAndExit(err error) {
    err = kh.ReportFailure([]string{err.Error()})
    if err != nil {
        log.Fatalln("error when reporting to kuberhealthy:", err)
    }
    os.Exit(0)
}

Sources: cmd/namespace-pod-check/main.go:159-166

Conclusion

Pod Health Checks serve as the frontline of Kuberhealthy's synthetic monitoring. By combining state-based checks (Pod Status), event-based checks (Pod Restarts), and workflow-based checks (Namespace Pod Check), the system provides a holistic view of cluster health that traditional metrics cannot capture alone. The decoupled architecture allows for easy extension through custom containers while maintaining strict security through UUID whitelisting.