Infrastructure & Storage Checks
Relevant source files
The following files were used as context for generating this wiki page:
Infrastructure and Storage Checks in Kuberhealthy are synthetic monitoring components designed to verify the continuous availability and performance of critical Kubernetes cluster subsystems. These checks go beyond simple health probes by replicating real-world workloads to identify underlying issues in container runtimes, node communication, image registries, and cloud infrastructure integrations.
The scope of these checks includes verifying that nodes can pull images within latency thresholds, deployments can mount storage and pass traffic, and IAM integrations like KIAM are correctly intercepting cloud metadata requests. By simulating these processes, Kuberhealthy can detect failures in disk schedulers, CNI communication, and Docker daemon errors that standard Kubernetes health checks might miss.
Sources: README.md:1-25, deploy/helm/kuberhealthy/README.md:10-25
Image and Registry Connectivity
Kuberhealthy provides specialized checks to ensure that the container runtime on nodes can successfully interact with image registries. This includes verifying registry reachability and measuring the performance of image downloads to ensure they meet specified latency requirements.
Image Download Performance
The image-download-check utilizes the crane library to pull a specified image, save it to /dev/null, and measure the elapsed time. This helps identify network congestion or registry throttling that could impact pod startup times.
The diagram shows the logic flow for measuring registry performance.
Sources: cmd/image-download-check/main.go:53-85
Configuration Parameters
| Environment Variable | Description |
|---|---|
FULL_IMAGE_URL | The registry, image name, and tag to test. |
TIMEOUT_LIMIT | Maximum duration allowed for the image pull. |
Sources: cmd/image-download-check/main.go:20-25
Cloud Infrastructure Integration (KIAM)
For clusters running in AWS, the KIAM check verifies the integrity of the KIAM server and agent infrastructure. It ensures that metadata requests are correctly intercepted and that the assigned IAM roles have the necessary permissions to interact with AWS services.
Logic Flow
The check attempts to list Lambda configurations from AWS using a specified IAM role. If the request is successful and returns any configurations, the KIAM infrastructure is deemed healthy.
The sequence diagram illustrates how the KIAM check validates IAM role interception and AWS service connectivity.
Sources: cmd/kiam-check/README.md:1-15
Deployment and Service Verification
This check verifies the "full stack" of Kubernetes infrastructure by creating, scaling, and updating actual resources. It ensures that the cluster can handle lifecycle events like rolling updates and service routing.
Operational Sequence
- Cleanup: Removes orphaned resources from previous runs.
- Creation: Deploys a set number of replicas (default: 2) and a ClusterIP service.
- Validation: Performs an HTTP GET request to the service endpoint to verify traffic flow.
- Rolling Update: (Optional) Updates the container image and validates traffic again to ensure no dropped connections.
- Teardown: Deletes the resources and verifies successful removal.
Sources: cmd/deployment-check/README.md:1-50
Resource Configuration
| Setting | Default Value | Description |
|---|---|---|
CHECK_IMAGE | nginxinc/nginx-unprivileged:1.17.8 | Initial image used for the deployment. |
CHECK_DEPLOYMENT_REPLICAS | 2 | Number of pods to provision. |
CHECK_DEPLOYMENT_ROLLING_UPDATE | false | Whether to perform an image update during the check. |
CHECK_CONTAINER_PORT | 8080 | Port the container listens on for health checks. |
Sources: cmd/deployment-check/README.md:55-75
Storage and Node Functionality Detection
Kuberhealthy checks are specifically designed to catch infrastructure failures that do not necessarily lead to node NotReady states. Examples of issues detected in production environments include:
- Disk Scheduler Errors: Pods stuck in
ContainerCreating. - CNI Communication Failures: Pods stuck in
Terminating. - High IO Wait: Nodes unable to provision pods quickly enough.
- Docker Daemon Errors: Pods stuck in
Pending.
Sources: deploy/helm/kuberhealthy/README.md:15-25, README.md:100-115
Summary
Infrastructure & Storage Checks provide a proactive verification layer for the Kubernetes control plane and data plane. By integrating with AWS IAM (via KIAM), verifying registry performance, and testing the full deployment lifecycle, these checks ensure that the underlying infrastructure is capable of supporting production workloads. This synthetic approach identifies transient and "silent" failures that traditional monitoring may overlook.
Sources: README.md:10-15, deploy/helm/kuberhealthy/README.md:10-15