Troubleshooting Guide

Relevant source files

The following files were used as context for generating this wiki page:

• charts/agh2/templates/validations/index.yaml
• charts/agh3/templates/validations/index.yaml
• charts/playground/templates/validations/index.yaml
• charts/agh3/templates/captain/captain-deployment.yml
• charts/agh2/templates/captain/captain-deployment.yml
• charts/agh2/templates/db/postgres-init-configmap.yml
• charts/agh3/templates/db/postgres-init-configmap.yml
• charts/agh3/templates/hooks/post/sidecar-container-enabler/0-sidecar-container-feature-enabler.cm.yml
• charts/agh3/values.yaml
• charts/agh2/values.yaml
• charts/agh2/templates/core/core-deployment.yml

This guide provides information on diagnosing and resolving common issues encountered during the deployment and operation of the agh2, agh3, and playground Helm charts. It covers configuration validation errors, service initialization failures, and problems with dependency setup.

Configuration Validation Failures

The Helm charts include a pre-install/pre-upgrade validation step that checks for required configuration values in values.yaml. If a required value is missing based on the enabled features, the Helm operation will fail with a "Validation Failed" message, detailing which fields are missing.

This diagram illustrates the Helm installation flow with the validation step.

AGH3 Validation Checks

The following table lists the validation rules for the agh3 chart. The deployment will fail if a feature is enabled but its required fields are not provided.

Condition	Required Fields	Error Message
keygen.enabled is true	accountID, apiToken	Must provide when keygen.enabled is true
customRegistrySecret.enabled is true	customRegistrySecret.auth.registry, username, password	Must provide when customRegistrySecret.enabled is true
ingress.tls.enabled is true	ingress.tls.cert, key	Must provide when ingress.tls.enabled is true
db.connection.type is external	db.connection.host, port	Must provide when db.connection is external
rabbitmq.connection.type is external	rabbitmq.connection.host, port	Must provide when rabbitmq.connection is external
minio.connection.type is external	minio.connection.host, port	Must provide when db.connection is external
captain.enabled and captain.secret.oidc.enabled are true	captain.secret.oidc.url, clientID, clientSecret, realm	Must provide when captain.secret.oidc.enabled is true

Sources: charts/agh3/templates/validations/index.yaml:3-69

AGH2 Validation Checks

The following table lists the validation rules for the agh2 chart.

Condition	Required Fields	Error Message
keygen.enabled is true	accountID, apiToken	Must provide when keygen.enabled is true
customRegistrySecret.enabled is true	customRegistrySecret.auth.registry, username, password	Must provide when customRegistrySecret.enabled is true
ingress.tls.enabled is true	ingress.tls.cert, key	Must provide when ingress.tls.enabled is true
db.connection.type is external	db.connection.host, port	Must provide when db.connection is external
minio.connection.type is external	minio.connection.host, port	Must provide when db.connection is external

Sources: charts/agh2/templates/validations/index.yaml:3-46

Playground Validation Checks

The playground chart has specific validation for its backend service configuration.

Condition	Required Fields	Error Message
playground.service.backendRef.kind is ExternalService	playground.service.backendRef.endpoint	Must provide when playground.service.backendRef.kind is ExternalService
playground.service.backendRef.kind is Service	name, port, protocol	Must provide when playground.service.backendRef.kind is Service
playground.service.backendRef.kind is invalid	-	Invalid value for playground.service.backendRef.kind (available options: Service, ExternalService)

Sources: charts/playground/templates/validations/index.yaml:23-58

Service Initialization Issues

Several deployments, such as captain and core, use initContainers to ensure that their dependencies are available before the main application container starts. If a dependency is not ready, the pod will remain in the Init status. You can check the logs of the init containers to diagnose the problem.

Captain Pod Initialization (AGH3)

The captain deployment in agh3 waits for Redis, Minio, RabbitMQ, and PostgreSQL to become available.

This sequence diagram shows the startup process for the captain pod.

Common commands used in init containers:

• Redis: until redis-cli -h redis-master... ping; do echo waiting for redis; sleep 1; done
• PostgreSQL: until pg_isready -d $PG_CONN_STR; do echo waiting for postgresql; sleep 1; done
• Minio: until curl http://minio...; do echo ... waiting... && sleep 1; done;
• RabbitMQ: until curl -f --user $RABBITMQ_USERNAME:$RABBITMQ_PASSWORD $RABBITMQ_API_ENDPOINT ; do ... done

Sources: charts/agh3/templates/captain/captain-deployment.yml:32-86, charts/agh2/templates/captain/captain-deployment.yml:25-47

Database Initialization (DB Init) Problems

When using the internal PostgreSQL database (db.internal.enabled: true), a ConfigMap named db-init-scripts is created to initialize databases and roles. This is executed by the initdb script of the Bitnami PostgreSQL chart. Failures in this script can prevent the database from starting correctly.

The initialization process involves:

1. Creating roles and databases for various application modules (captain, core, etc.).
2. For agh2 only, downloading and importing an attack.sql dataset from a Keygen artifact URL.

This flowchart shows the logic of the init.sh script inside the db-init-scripts ConfigMap.

Potential Failure Points:

• agh2 Attack Dataset Download: If keygen.apiToken or keygen.accountID are incorrect or missing in values.yaml, the curl command to download attack.sql.tgz will fail.
  Sources: charts/agh2/templates/db/postgres-init-configmap.yml:13-16
• SQL Script Execution: The final psql command executes the generated scripts. If there are any SQL errors (e.g., due to incorrect passwords or user configurations), the database initialization will halt.
  Sources: charts/agh2/templates/db/postgres-init-configmap.yml:102-111, charts/agh3/templates/db/postgres-init-configmap.yml:87-94

AGH3: Sidecar Container Feature Gate

The agh3 chart requires the SidecarContainers Kubernetes feature gate to be enabled. To automate this for k3s distributions, a post-install/post-upgrade Helm hook is included.

This hook runs a script on a cluster node to check for and enable the feature gate.

This flowchart shows the logic of the feature enabler script.

Potential Issues:

• The script is designed specifically for k3s and will likely fail on other Kubernetes distributions.
• The kubectl debug command requires elevated privileges to run a privileged container and chroot into the host's filesystem. If the ServiceAccount for the job lacks these permissions, the script will fail.
• Modifying and restarting the core k3s.service is a sensitive operation. A failure during this process could impact the stability of the node.

Sources: charts/agh3/templates/hooks/post/sidecar-container-enabler/0-sidecar-container-feature-enabler.cm.yml:19-20