AGH3 Core Concept: Job Queuing with Kueue
Relevant source files
The following files were used as context for generating this wiki page:
The AGH3 Helm chart provides an optional integration with Kueue, a Kubernetes-native job queueing system. When enabled, Kueue manages the execution of jobs within the cluster, providing resource quota management and controlling concurrency. This is particularly important for managing the worker jobs spawned by AGH3's Action Custom Resources, ensuring that the system does not become overloaded with too many simultaneous jobs.
The Kueue integration is disabled by default and can be activated via the kueue.enabled flag in the Helm chart's values.yaml. The implementation defines the necessary Kueue resources (ClusterQueue, LocalQueue, and ResourceFlavor) to establish a queuing mechanism for Kubernetes Job resources within the AGH3 deployment namespace.
Sources: charts/agh3/values.yaml:400-401, charts/agh3/templates/kueue/kueue.yml:1
Architecture and Components
The job queuing architecture in AGH3 relies on three fundamental Kueue resources that are created when the feature is enabled. These components work together to manage job workloads.
Sources: charts/agh3/templates/kueue/kueue.yml
Component Relationship
The following diagram illustrates how the Kueue components are structured and related.
This diagram shows that Jobs are submitted to the LocalQueue within the release namespace, which then funnels them into the ClusterQueue for resource allocation and scheduling.
Sources: charts/agh3/templates/kueue/kueue.yml:1-32
ClusterQueue
The ClusterQueue is a cluster-scoped resource that manages a pool of resources for multiple namespaces. In AGH3, it is configured to manage jobs and sets a limit on how many can run concurrently.
- Name: The name is configurable via
{{ .Values.kueue.queueName.cluster }}, with a default ofagh3-cluster-pool. - Resource Management: It manages the
jobsresource type. - Concurrency: The
nominalQuotafor jobs is set by{{ .Values.kueue.concurrentJobs }}, which defaults to100. This means up to 100 jobs can be scheduled concurrently through this queue.
# Source: charts/agh3/templates/kueue/kueue.yml:2-13
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: {{ .Values.kueue.queueName.cluster }}
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["jobs"]
flavors:
- name: "default-flavor"
resources:
- name: "jobs"
nominalQuota: {{ .Values.kueue.concurrentJobs }}
Sources: charts/agh3/templates/kueue/kueue.yml:2-13, charts/agh3/values.yaml:404-408
LocalQueue
The LocalQueue is a namespace-scoped resource that acts as the entry point for jobs within the AGH3 namespace. All jobs created in this namespace that are subject to queuing are directed to this LocalQueue.
- Name: The name is configurable via
{{ .Values.kueue.queueName.local }}, with a default ofagh3-local-pool. - Cluster Link: It is explicitly linked to the
ClusterQueuedefined by{{ .Values.kueue.queueName.cluster }}. This creates the pathway for jobs to move from the local namespace to the cluster-wide resource pool.
# Source: charts/agh3/templates/kueue/kueue.yml:17-25
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: {{ .Release.Namespace }}
name: {{ .Values.kueue.queueName.local }}
spec:
clusterQueue: {{ .Values.kueue.queueName.cluster }}
Sources: charts/agh3/templates/kueue/kueue.yml:17-25, charts/agh3/values.yaml:404-408
ResourceFlavor
The ResourceFlavor provides a way to differentiate types of resources. The AGH3 chart defines a simple default-flavor which is consumed by the ClusterQueue. This setup does not differentiate between types of jobs but provides the necessary link for the queuing mechanism to function.
Sources: charts/agh3/templates/kueue/kueue.yml:29-32
Configuration
The Kueue integration is configured under the kueue key in the values.yaml file. The feature is disabled by default.
| Parameter | Description | Default Value |
|---|---|---|
kueue.enabled | Enables or disables the creation of Kueue resources. | false |
kueue.queueName.cluster | The name of the ClusterQueue. | agh3-cluster-pool |
kueue.queueName.local | The name of the LocalQueue within the release namespace. | agh3-local-pool |
kueue.concurrentJobs | The maximum number of jobs that can run concurrently. This sets the nominalQuota. | 100 |
Sources: charts/agh3/values.yaml:400-408
To enable Kueue, set kueue.enabled to true in your Helm values:
# Example values.yaml override
kueue:
enabled: true
concurrentJobs: 50 # Optional: override the default concurrency
Job Lifecycle with Kueue
When Kueue is enabled, jobs created by AGH3 components (such as the workers for Action CRDs) are intercepted and managed by the queuing system.
The diagram below shows the flow of an Action from creation to its corresponding job being executed under Kueue's control.
This sequence demonstrates how Kueue enforces the concurrentJobs limit, preventing the cluster from being overwhelmed by Action worker jobs. The Action CRD itself defines the specifications for the worker containers.
Sources: charts/agh3/crds/actions.crds.yaml:10-13, 85-88, charts/agh3/templates/kueue/kueue.yml, charts/agh3/values.yaml:400-408, charts/agh3/templates/captain/captain-deployment.yml
Summary
The Kueue integration in the AGH3 Helm chart is a powerful, optional feature for managing job execution at scale. By leveraging ClusterQueue and LocalQueue resources, it provides a centralized mechanism to control job concurrency, ensuring system stability and fair resource allocation. Administrators can easily enable and configure this feature through the chart's values.yaml to suit their cluster's capacity and workload demands.
Sources: charts/agh3/values.yaml:400-408, charts/agh3/templates/kueue/kueue.yml