Master Calculation & Leader Election

Relevant source files

The following files were used as context for generating this wiki page:

• pkg/masterCalculation/masterCalculation.go
• pkg/masterCalculation/masterCalculation_test.go
• cmd/kuberhealthy/main.go
• cmd/kuberhealthy/kuberhealthy.go
• pkg/metrics/exporter.go
• README.md

Master Calculation and Leader Election is a core architectural component of Kuberhealthy that ensures only one instance of the operator acts as the "master" in multi-pod deployments. This mechanism is critical for preventing duplicate check executions and ensuring consistent reporting of cluster health state across a distributed environment. Sources: pkg/masterCalculation/masterCalculation.go:1-5, cmd/kuberhealthy/kuberhealthy.go:810-815

In a standard deployment, Kuberhealthy supports multiple replicas for failover capacity. However, while all pods can serve the status page, only the master pod is responsible for provisioning checker pods, scheduling KuberhealthyJobs, and performing the reaping of obsolete resources. Sources: deploy/helm/kuberhealthy/README.md:65-67, cmd/kuberhealthy/kuberhealthy.go:210-230

Architecture and Logic

The leader election process in Kuberhealthy is deterministic and relies on the Kubernetes API to list active Kuberhealthy pods. Instead of using complex consensus algorithms like Paxos or Raft, Kuberhealthy uses an alphabetical sorting strategy based on pod names to identify the master. Sources: pkg/masterCalculation/masterCalculation.go:50-65

Deterministic Master Selection

The selection process follows these steps:

1. Pod Discovery: Query the Kubernetes API for all pods in the current namespace with the label app=kuberhealthy and a status phase of Running. Sources: pkg/masterCalculation/masterCalculation.go:44-47
2. Naming Collection: Extract the names of all discovered pods into a string slice. Sources: pkg/masterCalculation/masterCalculation.go:53-56
3. Alphabetical Sort: Sort the list of pod names alphabetically. Sources: pkg/masterCalculation/masterCalculation.go:64
4. Selection: The pod at index 0 (the first in alphabetical order) is designated as the master. Sources: pkg/masterCalculation/masterCalculation.go:65

The diagram above shows the logic used by an individual Kuberhealthy pod to determine if it should assume the master role. Sources: pkg/masterCalculation/masterCalculation.go:41-94

Master Monitoring and State Management

Kuberhealthy pods continuously monitor the cluster state to handle failover scenarios. If the current master pod is deleted or crashes, the remaining pods recalculate the master, and the next pod in alphabetical order assumes the role. Sources: cmd/kuberhealthy/kuberhealthy.go:765-780

Monitoring Components

Component	Responsibility	Source
masterStatusWatcher	Watches for Kubernetes pod events with label app=kuberhealthy. Triggers a recalculation on any pod change.	cmd/kuberhealthy/kuberhealthy.go:765-805
masterMonitor	Evaluates the "upcoming" master state and manages the transition (becoming or losing master status).	cmd/kuberhealthy/kuberhealthy.go:810-845
lastMasterChangeTime	A timestamp used to implement a settling period, preventing rapid flapping during cluster updates.	cmd/kuberhealthy/main.go:27, cmd/kuberhealthy/kuberhealthy.go:825-828

Failover Sequence

When a master transition occurs, Kuberhealthy uses Go channels to notify the main control loop to stop or start check execution. Sources: cmd/kuberhealthy/kuberhealthy.go:215-225

The sequence diagram illustrates how a pod transitions from standby to master status after a pod deletion event. Sources: cmd/kuberhealthy/kuberhealthy.go:765-845

Master Responsibilities

The master pod performs specific administrative tasks that standby pods ignore. This ensures that cluster-wide operations are centralized. Sources: cmd/kuberhealthy/kuberhealthy.go:210-230

• Check Scheduling: Running the runCheck loops for all configured khcheck resources. Sources: cmd/kuberhealthy/kuberhealthy.go:945-950
• Job Execution: Triggering khjob workloads when new resources are added to the cluster. Sources: cmd/kuberhealthy/kuberhealthy.go:635-645
• Resource Reaping: Running the khStateResourceReaper to clean up state objects that no longer have corresponding checks. Sources: cmd/kuberhealthy/kuberhealthy.go:275-285
• Metric Forwarding: While all pods can export Prometheus metrics, the master calculates the authoritative state. Sources: pkg/metrics/exporter.go:45-55

Configuration and Debugging

Several configuration options and environment variables influence how the leader election behaves. Sources: cmd/kuberhealthy/main.go:20-60

Option	Type	Description
POD_NAME	Env Var	Required. Used by the pod to identify itself in the pod list.
POD_NAMESPACE	Env Var	Defines the scope of the pod listing for master calculation.
--forceMaster	Flag	Debugging flag that forces the current pod to always report as master.

Forced Master Mode

For development and debugging, a forced master mode can be enabled. When enableForceMaster is true, the IAmMaster function skips all logic and immediately returns true. Sources: pkg/masterCalculation/masterCalculation.go:74-78

// IAmMaster determines if the executing pod is the cluster master or not
func IAmMaster(client *kubernetes.Clientset) (bool, error) {

	// if we are in debug enable master always, then just return true
	if enableForceMaster {
		return true, nil
	}
    // ... logic continues
}

Sources: pkg/masterCalculation/masterCalculation.go:72-80

Summary

The Master Calculation and Leader Election system provides a robust, low-dependency method for coordinating multiple Kuberhealthy instances. By leveraging the Kubernetes API as a source of truth and employing deterministic alphabetical sorting, the system ensures that synthetic monitoring remains continuous and non-redundant even during pod lifecycle events or cluster maintenance. Sources: pkg/masterCalculation/masterCalculation.go, cmd/kuberhealthy/kuberhealthy.go:195-245