MiqWorker & Process Management

Relevant source files

The following files were used as context for generating this wiki page:

• app/models/miq_worker.rb
• app/models/miq_worker/container_common.rb
• app/models/miq_server/status_management.rb
• app/models/miq_task.rb
• app/models/miq_request_task.rb
• spec/models/miq_worker/runner_spec.rb
• spec/models/miq_server/worker_management/systemd_spec.rb

MiqWorker is the base class for all background processes in ManageIQ, responsible for executing specific roles such as event catching, metrics collection, and UI rendering. The process management system ensures that the correct number of workers are running, monitors their health via heartbeats, and manages their lifecycle across different environments (Standard Linux, Systemd, or Podified/Containerized).

The management logic is distributed between the MiqWorker model, which represents the worker's state in the database, and the MiqServer, which acts as an orchestrator for the workers in its zone.

Worker Lifecycle and States

Workers transition through a defined set of statuses from creation to termination. These states allow the MiqServer to synchronize the active processes with the desired configuration.

Status Definitions

Status	Description
creating	The worker record is being initialized in the database.
starting	The process has been spawned but has not yet heartbeated.
started	The process is running and successfully heartbeating.
ready	The worker is prepared to accept work items.
working	The worker is currently processing a task from the MiqQueue.
stopping	A stop signal has been sent; the worker is finishing current tasks.
stopped	The process has terminated gracefully.
killed	The process was terminated forcefully (SIGKILL).
aborted	The process disappeared or failed unexpectedly.

Sources: app/models/miq_worker.rb:27-35, app/models/miq_worker.rb:194-202

State Transitions

The following diagram illustrates the typical lifecycle of a worker process:

Process Spawning and Environments

ManageIQ supports multiple deployment architectures, and the MiqWorker class abstracts the spawning logic based on the environment.

1. Standard Spawn (Linux)

In standard environments, workers are spawned using Kernel.spawn. The command line is constructed using a GUID and optional EMS ID. Environment variables like BUNDLER_GROUPS are passed to ensure the worker loads only necessary dependencies.
Sources: app/models/miq_worker.rb:298-316, spec/models/miq_worker/runner_spec.rb:147-154

2. Systemd Integration

When Systemd is supported, workers are managed as Systemd services. The system_uid of the worker is mapped to the unit name (e.g., manageiq-generic@<guid>.service). The server interacts with the Systemd Manager via DBus to start, stop, or reset failed units.
Sources: app/models/miq_worker.rb:288-290, spec/models/miq_server/worker_management/systemd_spec.rb:13-33

3. Containerized (Podified)

In containerized environments (Kubernetes/OpenShift), the server creates Deployment objects. The ContainerCommon module handles the generation of deployment definitions, including image selection, environment variables, and resource constraints.
Sources: app/models/miq_worker/container_common.rb:6-25, spec/models/miq_worker/container_common_spec.rb:18-40

Health Monitoring and Resource Management

Heartbeat and Timeout

Workers are required to update their last_heartbeat periodically. If a worker in a stopping state fails to heartbeat within the stopping_timeout, it is considered "stopping for too long" and may be forcefully killed.
Sources: app/models/miq_worker.rb:364-370, spec/models/miq_worker/runner_spec.rb:14-18

Resource Constraints

In containerized environments, workers can have CPU and memory limits/requests enforced. These are derived from worker settings:

• Memory Threshold: Maximum memory a worker can consume before the monitor intervenes.
• CPU Threshold: Percentage of CPU allocated to the worker.

def resource_constraints
  return {} unless Settings.server.worker_monitor.enforce_resource_constraints
  mem_limit = self.class.worker_settings[:memory_threshold]
  cpu_limit = self.class.worker_settings[:cpu_threshold_percent]
  # ... formatting logic ...
end

Sources: app/models/miq_worker/container_common.rb:64-81, spec/models/miq_worker/container_common_spec.rb:185-220

Status Updates

The MiqServer periodically invokes status_update on workers. For non-podified environments, this uses MiqProcess.processInfo(pid) to collect:

• Memory usage and size.
• Proportional Set Size (PSS) and Unique Set Size (USS).
• CPU time and percentage.
• OS priority.

Sources: app/models/miq_worker.rb:406-422, app/models/miq_server/status_management.rb:23-29

Task and Request Integration

Workers are the execution engines for MiqTask and MiqRequestTask. When a worker is destroyed, it performs a cleanup of its active messages.

Message Cleanup

If a worker is deleted while handling messages:

1. Active Messages: Messages in dequeue state are moved to an error state.
2. Linked Tasks: Any MiqTask associated with the active message is updated to a Finished state with a status of Error.
3. Ready Messages: Messages not yet started are nullified (handler removed) so other workers can process them.

Sources: app/models/miq_worker.rb:387-404, app/models/miq_request_task.rb:108-115, app/models/miq_task.rb:130-138

Sequence of Task Execution

Configuration and Synchronization

The number of running workers is synchronized based on the worker_settings.

Worker Synchronization Logic

The sync_workers method calculates the difference between desired (from config) and current (alive in DB).

• If desired > current: The server attempts to start new workers if enough resources exist.
• If desired < current: Excess workers are sent a stop signal.

Sources: app/models/miq_worker.rb:128-147, spec/models/miq_worker_spec.rb:22-28

Settings Hierarchy

Worker settings are fetched with a specific inheritance pattern:
worker_base -> queue_worker_base -> specific_worker_type -> provider_specific_worker.
Values like memory_threshold can be overridden at any level of this hierarchy.
Sources: app/models/miq_worker.rb:174-192, spec/models/miq_worker_spec.rb:113-145

Summary

MiqWorker and Process Management form the backbone of the ManageIQ distributed architecture. By abstracting the complexities of process lifecycle, health monitoring, and environment-specific spawning, the system provides a robust platform for executing asynchronous tasks. The tight integration with MiqServer ensures that the worker fleet remains healthy and scaled appropriately to the system's requirements.