Health Checking

Relevant source files

The following files were used as context for generating this wiki page:

• internal/server/health_check.go
• internal/server/service.go
• internal/cmd/deploy.go
• internal/server/health_check_backoff_test.go
• internal/server/target_test.go
• internal/server/health_check_test.go

Health checking in kamal-proxy is a background mechanism designed to ensure that upstream targets are ready and capable of serving traffic before they are included in the load-balancing pool. It handles the transition from a "cold wake" (where a container process exists but the application is not yet listening) to a steady state by using an adaptive probing strategy.

The system is built around the HealthCheck struct, which periodically probes a specific URL endpoint using HTTP GET requests. It reports the results to a HealthCheckConsumer (typically a Target), which then updates its internal state to reflect whether it is healthy or unhealthy.

Sources: internal/server/health_check.go:1-40, internal/server/health_check_backoff_test.go:37-43

Architecture and Components

The health checking system consists of a controller that manages an independent background goroutine for each target.

HealthCheck Struct

The HealthCheck struct maintains the state of a single probing cycle for a target. It includes the target endpoint, interval, timeout, and a reference to the consumer that receives updates.

Field	Description
consumer	An interface (HealthCheckConsumer) notified upon completion of a probe.
endpoint	The *url.URL to probe (e.g., http://target:port/up).
interval	The steady-state duration between probes.
timeout	Maximum duration allowed for a single probe request.
becameHealthy	An atomic boolean that latches true after the first successful probe.

Sources: internal/server/health_check.go:30-46

Probing Lifecycle

When a HealthCheck is initiated, it enters a loop that performs an immediate probe followed by a series of retries. The logic distinguishes between the initial discovery phase and the steady-state monitoring phase.

This diagram shows the transition from the aggressive initial backoff to the steady-state interval once a target is confirmed healthy.
Sources: internal/server/health_check.go:73-98

Backoff and Timing Logic

kamal-proxy implements a specific backoff strategy to minimize the wait time during container deployment.

Exponential Backoff

A target is often not listening the moment a container starts. To avoid waiting for a full check interval (e.g., 1 second) if the target becomes ready in 100ms, the proxy starts with a small initialHealthCheckDelay of 50ms.

• Initial Delay: 50ms.
• Scaling: If a probe fails, the delay doubles (delay * 2).
• Cap: The delay is always capped by the user-configured interval.
• Reset: Once becameHealthy is latched, the delay settles permanently to the configured interval.

Sources: internal/server/health_check.go:27-28, 90-98, internal/server/health_check_backoff_test.go:44-55

Probe Execution

Each probe is an HTTP GET request. The proxy sets a specific User-Agent: kamal-proxy. A successful probe requires a status code in the 2xx range (200-299).

req.Header.Set("User-Agent", healthCheckUserAgent)

resp, err := http.DefaultClient.Do(req)
// ... error handling ...
if resp.StatusCode < 200 || resp.StatusCode > 299 {
    hc.reportResult(false, fmt.Errorf("%w (%d)", ErrorHealthCheckUnexpectedStatus, resp.StatusCode))
    return
}
hc.reportResult(true, nil)

Sources: internal/server/health_check.go:107-133

Configuration

Health checking is configured primarily during service deployment via CLI flags or the ServiceOptions and TargetOptions structures.

Default Values

Parameter	Default Value	Description
Path	/up	The URL path to probe.
Interval	1s	Time between probes in steady state.
Timeout	5s	Timeout for the HTTP request.
Port	0	Defaults to match the target's service port.

Sources: internal/server/service.go:60-64

Deployment Flags

The deploy command provides several flags to customize these parameters per target.

• --health-check-interval: Sets the steady-state interval.
• --health-check-timeout: Sets the request timeout.
• --health-check-path: Customizes the endpoint path (e.g., /health).
• --health-check-host: Sets a custom Host header for the health check request.

Sources: internal/cmd/deploy.go:52-57

Integration with Targets

The Target component is the primary consumer of health check results. When a health check completes, the Target updates its state, which affects its eligibility for selection by the LoadBalancer.

This sequence illustrates how the Target initiates and reacts to the HealthCheck cycles.
Sources: internal/server/target_test.go:582-595, internal/server/health_check.go:136-144

Health Check URL Construction

The URL for probing is constructed by combining the target's address with the configured health check port and path. If the health check port is 0, it uses the target's primary port.

Sources: internal/server/target_test.go:616-645

Summary

The Health Checking system ensures high availability by aggressively monitoring new targets with a 50ms-start exponential backoff and maintaining a steady-state heartbeat once healthy. By integrating directly with the Target and LoadBalancer, it prevents traffic from being routed to uninitialized or failing upstreams.