Object Storage (Minio)

Relevant source files

The following files were used as context for generating this wiki page:

• charts/agh3/values.yaml
• charts/agh2/values.yaml
• charts/agh3/templates/minio/agh-minio-users-secret.yml
• charts/agh2/templates/minio/agh-minio-users-secret.yml
• charts/agh3/Chart.yaml
• charts/agh3/templates/captain/captain-deployment.yml
• charts/agh2/templates/core/core-deployment.yml
• charts/agh3/templates/minio/agh-minio-cm.yml
• charts/agh3/templates/captain/captain-minio-secret.yml
• charts/agh2/templates/core/core-minio-secret.yml
• charts/agh3/templates/parser/parser-minio-secret.yml
• charts/agh3/templates/validations/index.yaml

Minio serves as the primary object storage solution within the AGH2 and AGH3 Helm charts. It is responsible for storing various data artifacts required by different application components, such as reports, integration data, and machine learning assets. The system is designed to be flexible, supporting both an internally managed Minio instance deployed as a sub-chart and the ability to connect to an external, pre-existing Minio service.

When deployed internally, the Helm chart automates the provisioning of Minio, including the creation of buckets, users, and access policies tailored to the needs of each microservice. This ensures that components like Captain, Core, and Parser have the necessary, yet restricted, access to object storage from the moment they are deployed. This automated setup simplifies deployment and enhances security by enforcing the principle of least privilege for each service.

Architecture and Deployment

The Minio integration can be configured in one of two modes: internal or external. This is controlled by the minio.connection.type value in the values.yaml file for both AGH2 and AGH3 charts.

• Internal Mode: If minio.connection.type is set to internal, the chart deploys Minio as a dependency using the Bitnami Minio chart. This is the default and recommended setup for a self-contained environment.
• External Mode: If set to external, the chart will not deploy a Minio instance. Instead, it will use the connection details (host, port) provided in the configuration to connect to an existing Minio service.

Sources: charts/agh3/values.yaml:294-306, charts/agh2/values.yaml:194-206, charts/agh3/Chart.yaml:23-25

The following diagram illustrates the deployment decision logic.

Connection Configuration

The primary connection settings for Minio are defined under the minio.connection key.

Parameter	Description	AGH3 Default	AGH2 Default
type	Deployment type (internal or external).	internal	internal
host	Minio host address (for external type).	database.example.com	database.example.com
port	Minio host port (for external type).	9000	9000
user	Default Minio user.	argushack	argushack
password	Default Minio password.	""	""

Sources: charts/agh3/values.yaml:293-311, charts/agh2/values.yaml:193-211, charts/agh3/templates/validations/index.yaml:57-67

Automated Provisioning (Internal Mode)

When using the internal Minio deployment, the chart can automatically provision buckets, users, and policies upon initialization. This is enabled by default with minio.provisioning.enabled: true.

Sources: charts/agh3/values.yaml:342-343, charts/agh2/values.yaml:244-245

Bucket Provisioning (AGH3)

The AGH3 chart creates several buckets, some with versioning enabled, to store different types of data.

Bucket Name	Versioning Enabled	Purpose
intelli-bridge	true	Data for the Intelli-Bridge feature.
embedding	true	Storage for embedding models or data.
packet	false	Storage for network packet captures.
integration	false	Data related to third-party integrations.
report-setting	false	Settings for report generation.
review	true	Data for review processes.

Sources: charts/agh3/values.yaml:346-357

Policy and User Provisioning

The charts define specific IAM policies and create users for various microservices, assigning them the appropriate policies to enforce access control.

AGH3 Policies and Users

In AGH3, three distinct policies are created to grant scoped permissions to different services.

Policy Name	Permissions (Effect: Allow)	Assigned To User
agh-controller-policy	s3:GetBucketLocation, s3:ListBucket, s3:GetObject on arn:aws:s3:::*	executor-minio-user
agh-ai-policy	s3:* on intelli-bridge, embedding, integration, review buckets	parser-minio-user
agh-packet-policy	s3:* on packet bucket	packet-minio-user

Sources: charts/agh3/values.yaml:361-410, charts/agh3/templates/minio/agh-minio-users-secret.yml:38-41, charts/agh3/templates/minio/agh-minio-users-secret.yml:55-58, charts/agh3/templates/minio/agh-minio-users-secret.yml:72-75

The captain service user (capt-minio-user) is granted a general readwrite policy.

Sources: charts/agh3/templates/minio/agh-minio-users-secret.yml:19-20

AGH2 Policies and Users

The AGH2 chart defines a simpler policy structure.

Policy Name	Permissions (Effect: Allow)	Assigned To User
agh-core-policy	s3:GetBucketLocation, s3:ListBucket, s3:GetObject on arn:aws:s3:::*	core-minio-user

Sources: charts/agh2/values.yaml:249-256, charts/agh2/templates/minio/agh-minio-users-secret.yml:32-33

The captain service user (capt-minio-user) is granted a general readwrite policy.

Sources: charts/agh2/templates/minio/agh-minio-users-secret.yml:19-20

The following diagram shows the relationship between users, policies, and buckets in the AGH3 chart.

Secrets Management

Credentials for Minio are managed through Kubernetes Secrets. Separate secrets are created for each service, containing the specific user and password for that service to connect to Minio.

Service-Specific Secrets

A dedicated secret is created for each service that requires Minio access. For example, the captain service has a secret named capt-minio-secret.

# Source: charts/agh3/templates/captain/captain-minio-secret.yml
{{- if and .Values.captain.enabled (and .Values.captain.secret.enabled .Values.captain.secret.minio.enabled) }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.captain.secret.minio.secretName }}
  labels:
    {{- include "AGH3.labels" . | nindent 4 }}
stringData:
  capt-minio-user: {{ (required "captain.secret.minio.user is required" .Values.captain.secret.minio.user) | quote }}
  capt-minio-password: {{
    (
      default
        .Values.captain.secret.minio.password
        (
          # ... password generation logic ...
        )
    ) | substr 0 40 | quote
  }}
{{- end }}

This snippet shows the template for creating the Captain service's Minio secret. Similar templates exist for core, parser, etc.
Sources: charts/agh3/templates/captain/captain-minio-secret.yml, charts/agh2/templates/core/core-minio-secret.yml, charts/agh3/templates/parser/parser-minio-secret.yml

User Provisioning Secret

A special secret, agh-minio-users-secret, is created to feed the Minio provisioning job. This secret contains definitions for multiple users, their passwords, and the policies they should be assigned.

# Source: charts/agh3/templates/minio/agh-minio-users-secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: agh-minio-users-secret
type: Opaque
stringData:
  # ... captain user definition ...
  {{ .Values.controller.secret.minio.user }}: |
    username={{ .Values.controller.secret.minio.user }}
    password={{ ... }}
    disabled=false
    {{- with index .Values.minio.provisioning.policies 0 }}
    policies={{ .name }}
    {{- end }}
    setPolicies=false
  # ... other user definitions ...

This secret is consumed by the Minio chart's provisioning job to create all necessary application users at once.
Sources: charts/agh3/templates/minio/agh-minio-users-secret.yml, charts/agh2/templates/minio/agh-minio-users-secret.yml

Service Integration

Application services are configured via environment variables to connect to Minio. These variables are populated from the secrets created during deployment.

The following diagram shows how the captain service gets its configuration to connect to Minio.

An example from the captain deployment in AGH3 shows how the environment variables are set:

# Source: charts/agh3/templates/captain/captain-deployment.yml:148-161
          env:
            # ... other envs
            - name: MINIO_URL
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.minio.secret.secretName }}
                  key: url
            - name: MINIO_USER
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.captain.secret.minio.secretName }}
                  key: capt-minio-user
            - name: MINIO_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.captain.secret.minio.secretName }}
                  key: capt-minio-password

Sources: charts/agh3/templates/captain/captain-deployment.yml:148-161, charts/agh2/templates/core/core-deployment.yml:68-79

Webhook Notifications (AGH3)

In the AGH3 chart, Minio is configured to send webhook notifications for object events. This is managed by the agh-minio-cm ConfigMap, which sets environment variables for the Minio instance.

Environment Variable	Value
MINIO_NOTIFY_WEBHOOK_ENABLE_PRIMARY	on
MINIO_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY	http://captain.<namespace>.svc.cluster.local:8080/internal/webhook/minio

This configuration instructs Minio to send a POST request to the captain service's internal webhook endpoint whenever an event occurs, allowing for reactive processing of stored objects.

Sources: charts/agh3/templates/minio/agh-minio-cm.yml

Summary

The Minio object storage system is a cornerstone of the data persistence strategy in both AGH2 and AGH3. Its integration via Helm allows for a highly configurable deployment, supporting both embedded and external instances. The automated provisioning of buckets, users, and fine-grained policies in the default internal mode significantly streamlines setup and enhances security. Services are decoupled from storage configuration through the use of Kubernetes secrets, promoting a clean and maintainable architecture. The addition of webhook notifications in AGH3 further enables event-driven workflows, making the object store an active component of the application's logic.