AGH3 Core Concept: Controller & ActionLoop
Relevant source files
The following files were used as context for generating this wiki page:
The Controller and ActionLoop components form the backbone of AGH3's asynchronous task execution engine. This system is built around a Kubernetes-native operator pattern, using a Custom Resource Definition (CRD) to define and manage tasks. The Controller acts as the operator, watching for Action resources and translating them into executable Kubernetes Pods. It manages the lifecycle of these tasks within the Kubernetes cluster.
The ActionLoop is a separate, scalable backend service that consumes messages produced by the Controller upon task completion. It is responsible for handling the business logic that follows a task's execution, such as updating databases, storing artifacts, or triggering subsequent workflows. This decoupled, event-driven architecture provides a robust and scalable mechanism for orchestrating complex operations within the AGH3 platform.
Architecture Overview
The Controller and ActionLoop work in concert with the Kubernetes API server and other backend services like RabbitMQ, MinIO, and PostgreSQL to execute and process tasks. The Action CRD serves as the primary interface for initiating these tasks.
The following diagram illustrates the high-level interaction between these components.
AGH3 Controller
The AGH3 Controller is a Kubernetes controller that implements the operator pattern. Its main function is to watch for Action custom resources and manage their lifecycle. When a new Action is created, the Controller creates a corresponding worker Pod to execute the task defined in the Action's specification.
Sources: charts/agh3/templates/controller/controller-deployment.yml, charts/agh3/crds/actions.crds.yaml
Deployment
The Controller is deployed as a Kubernetes Deployment resource. It runs as a single pod that requires access to the Kubernetes API to manage Action resources and their associated pods.
Key aspects of its deployment include:
- An
initContainerthat waits for RabbitMQ to be available before the main controller container starts. - A service account (
controller-sa) which is granted permissions to interact with cluster resources.
Sources: charts/agh3/templates/controller/controller-deployment.yml:5-34
Configuration
The Controller's behavior is configured through environment variables, which are populated from values.yaml, secrets, and Kubernetes's Downward API.
| Environment Variable | Description | Source Value / Secret |
|---|---|---|
KUEUE_NAME | The name of the Kueue local queue, if Kueue is enabled. | {{ .Values.kueue.queueName.local }} |
OTEL_COLLECTOR_URL | The endpoint for the OpenTelemetry collector for tracing, if enabled. | grafana-tempo-distributor... |
SA_NAME | The name of the service account the controller's worker pods will use. | controller-sa |
NAMESPACE | The Kubernetes namespace where the controller is deployed. | Downward API (metadata.namespace) |
MINIO_URL | The URL for the Minio object storage service. | minio-secret |
MINIO_USER | The user for authenticating with Minio. | executor-minio-secret |
MINIO_PASSWORD | The password for the Minio user. | executor-minio-secret |
RABBITMQ_QUEUE | The name of the RabbitMQ queue to publish results to. | action_crd_result |
RABBITMQ_Conn | The connection string for RabbitMQ. | agh-rabbitmq-secret-connection |
REGISTRY_URL | The URL of the container image registry. | controller.env.REGISTRY_URL or global.imageRegistry |
Sources: charts/agh3/templates/controller/controller-deployment.yml:71-118, charts/agh3/values.yaml:1003-1033
AGH3 ActionLoop
The ActionLoop is a backend service that processes the outcomes of executed Actions. It listens to a specific RabbitMQ queue for messages, which are published by the Controller when an Action's worker pod completes. This component is responsible for executing follow-up business logic, such as updating database records, storing artifacts in Minio, or sending notifications.
Sources: charts/agh3/templates/actionloop/actionloop-deployment.yml
Deployment
ActionLoop is deployed as a Kubernetes Deployment and can be scaled by adjusting the replicaCount to handle varying loads. Its initContainers ensure that dependencies like Redis, RabbitMQ, and PostgreSQL are ready before the application starts.
Sources: charts/agh3/templates/actionloop/actionloop-deployment.yml:5-10, 15-68
Configuration
ActionLoop is configured via environment variables that connect it to various backend services.
| Environment Variable | Description | Source Value / Secret |
|---|---|---|
IMAGE_REGISTRY_URL | The URL of the container image registry. | {{ .Values.global.imageRegistry }} |
REDIS_URL | The connection URL for the Redis instance. | redis-master.$(NAMESPACE).svc.cluster.local:6379 |
REDIS_PASSWORD | The password for Redis authentication. | agh-redis-secret |
DB_Conn | The connection string for the PostgreSQL database. | capt-db-secret |
RABBITMQ_QUEUE | The RabbitMQ queue from which to consume messages. | {{ .Values.rabbitmq.queueName }} (action_crd_result) |
RABBITMQ_Conn | The connection string for RabbitMQ. | agh-rabbitmq-secret-connection |
SMTP_HOST / SMTP_PORT | SMTP server details for sending emails, if enabled. | postfix.$(NAMESPACE).svc.cluster.local |
MINIO_URL / USER / PASSWORD | Credentials for Minio object storage. | minio-secret, capt-minio-secret |
Sources: charts/agh3/templates/actionloop/actionloop-deployment.yml:90-142, charts/agh3/values.yaml:810-833
Health and Readiness
The ActionLoop deployment includes liveness and readiness probes to ensure the service is healthy and ready to receive traffic.
- Liveness Probe:
GET /healthzon port8080 - Readiness Probe:
GET /readyzon port8080
Sources: charts/agh3a/templates/actionloop/actionloop-deployment.yml:143-158
Action Custom Resource (CRD)
The Action CRD, with the group agh.lkc-lab.com, is the core API for defining and executing tasks. It provides a declarative way to specify container-based workloads. The Controller monitors these resources and acts upon them. The CRD has evolved, with v3alpha2 being the current storage version.
Sources: charts/agh3/crds/actions.crds.yaml:5-11, 1024-1027
Key spec Fields
The spec of an Action resource defines the desired state of the task.
| Field | Type | Description |
|---|---|---|
spec.containers | array | Defines the primary worker container(s), including image, command, arguments, environment variables, and volume mounts. This is a required field. |
spec.initContainers | array | Defines one or more init containers that run to completion before the main containers start. |
spec.historyID | string | An identifier used to associate a group of related actions. This is a required field. |
spec.isRoot | boolean | A flag to identify the action as a root action in a potential workflow. Defaults to false. |
spec.packet | object | If enabled, extends the action with packet capture capabilities. |
spec.judgement | object | A special post-execution hook that runs only if the main action succeeds. It has its own container definition. |
Sources: charts/agh3/crds/actions.crds.yaml:1032-1044, 1109-1113, 1170-1173, 1254-1262
Key status Fields
The status subresource reflects the observed state of the Action.
| Field | Type | Description |
|---|---|---|
status.activeStatus | string | The current state of the action, such as Pending, Running, Succeeded, or Failed. This field is updated by the Controller. |
status.worker | object | A reference to the Kubernetes Pod that is or was executing the action. |
Sources: charts/agh3/crds/actions.crds.yaml:1372-1376, 1377-1411
End-to-End Task Execution Flow
The following sequence diagram shows the complete lifecycle of an Action from creation to final processing by the ActionLoop.
Summary
The AGH3 Controller and ActionLoop provide a powerful, decoupled system for managing asynchronous tasks in a Kubernetes environment. The Controller acts as the "brain," translating declarative Action resources into running pods and managing their state. The ActionLoop serves as the "hands," consuming task results from a message queue to perform essential backend business logic. This separation of concerns, combined with the use of standard cloud-native technologies like Kubernetes CRDs and RabbitMQ, results in a flexible, scalable, and resilient task execution framework at the core of AGH3.