Writing Custom Checks

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• cmd/http-check/main.go
• pkg/checks/external/main.go
• cmd/kuberhealthy/kuberhealthy.go
• cmd/http-content-check/main.go
• clients/js/README.md
• cmd/test-check/README.md

Kuberhealthy allows for the creation of custom synthetic tests to verify that applications and Kubernetes clusters are functioning as expected. These checks are implemented as independent containers that run within the cluster, perform a specific verification logic, and report results back to the Kuberhealthy operator.

Custom checks can be written in any programming language. Kuberhealthy provides client libraries (such as Go and JavaScript) to simplify the process of reporting success or failure. Once a container image is created, it is deployed using a KuberhealthyCheck (khcheck) custom resource.

Sources: README.md, pkg/checks/external/main.go:53-70

External Check Lifecycle

The Kuberhealthy operator manages the lifecycle of custom checks by scheduling checker pods based on the configuration defined in the KuberhealthyCheck resource.

The diagram shows the interaction between the Kuberhealthy operator, the Kubernetes API, and the custom checker pod.
Sources: README.md, cmd/kuberhealthy/kuberhealthy.go:374-450

Key Lifecycle Steps

1. Scheduling: Kuberhealthy observes a KuberhealthyCheck and schedules a pod based on the provided podSpec.
2. Execution: The checker pod starts and executes its internal logic (e.g., hitting an API, checking a database).
3. Reporting: The pod sends a JSON payload to the Kuberhealthy status endpoint (/externalCheckStatus).
4. Persistence: Kuberhealthy stores the result in a KHState resource and updates Prometheus metrics.

Sources: README.md, pkg/checks/external/main.go:490-580

Reporting Results

Custom checks must report their status to Kuberhealthy via an HTTP POST request to the internal Kuberhealthy service. The operator provides environment variables to the checker pod to facilitate this communication.

Required Environment Variables

Kuberhealthy automatically injects the following environment variables into the checker pod:

Variable	Description
KH_REPORTING_URL	The URL where the check should report its status (Default: http://kuberhealthy.kuberhealthy.svc.cluster.local/externalCheckStatus).
KH_RUN_UUID	A unique identifier for the current run, used for de-duplication and security validation.
KH_CHECK_RUN_DEADLINE	Unix timestamp indicating when the check must complete its run.

Sources: pkg/checks/external/main.go:27-39, cmd/http-check/main.go:33-40

Reporting Payload

The results are reported using a JSON object containing the overall status and an optional list of error messages.

{
    "OK": false,
    "Errors": ["Failed to reach target API", "Timeout after 5s"]
}

Sources: cmd/kuberhealthy/kuberhealthy.go:945-960

Implementation Examples

Go Implementation

Using the checkclient package allows Go-based checks to report results easily.

package main

import (
    "github.com/kuberhealthy/kuberhealthy/v2/pkg/checks/external/checkclient"
)

func main() {
    // Perform custom check logic here
    err := doWork()
    if err != nil {
        checkclient.ReportFailure([]string{err.Error()})
        return
    }
    checkclient.ReportSuccess()
}

Sources: README.md, cmd/http-check/main.go:210-222

JavaScript Implementation

A JavaScript client is also available, supporting both synchronous and asynchronous reporting.

const kh = require("./kh-client");

try {
    // Perform check logic
    kh.ReportSuccess();
} catch (err) {
    kh.ReportFailure([err.message]);
}

Sources: clients/js/README.md

Configuration (KuberhealthyCheck)

The KuberhealthyCheck custom resource defines how the check should be executed, including the run interval and the pod specification.

Field	Description	Default
runInterval	How often the check should be executed.	15m
timeout	Maximum time allowed for the check pod to run.	15m
podSpec	Standard Kubernetes PodSpec defining the container image and environment.	N/A

Example Spec

apiVersion: comcast.github.io/v1
kind: KuberhealthyCheck
metadata:
  name: http-content-check
  namespace: kuberhealthy
spec:
  runInterval: 5m
  timeout: 10m
  podSpec:
    containers:
      - name: main
        image: kuberhealthy/http-content-check:v2.0.0
        env:
          - name: TARGET_URL
            value: "https://example.com"
          - name: TARGET_STRING
            value: "Example Domain"

Sources: cmd/http-content-check/main.go:20-30, cmd/test-check/README.md

Security and Validation

Kuberhealthy ensures that only authorized pods can report status for a specific check.

1. UUID Whitelisting: Kuberhealthy generates a KH_RUN_UUID for every run and stores it in the KHState. The reporting pod must include this UUID in its report.
2. Source IP Validation: If the UUID header is missing, Kuberhealthy validates the request based on the reporting pod's source IP address.
3. Annotations: Checker pods are labeled with kuberhealthy-check-name and annotated with comcast.github.io/check-name for validation during the reporting process.

Sources: cmd/kuberhealthy/kuberhealthy.go:815-885, pkg/checks/external/main.go:33-45

Summary

Writing custom checks in Kuberhealthy involves creating a containerized application that performs specific validation logic and reports the result via a simple JSON API. By utilizing the provided KuberhealthyCheck CRD, developers can integrate bespoke synthetic monitoring into their Kubernetes clusters with native Prometheus support and automated scheduling.

Sources: README.md, pkg/checks/external/main.go:580-600