Kubernetes Operations
Relevant source files
The following files were used as context for generating this wiki page:
Kubernetes Operations in the infrahub-ops-cli project provide automated backup, restore, environment detection, and troubleshooting bundle collection. The infrastructure relies on dedicated Kubernetes workloads that execute shell-out commands to the cluster via binary utilities. This system operates natively within GitOps workflows managed by platforms like ArgoCD or Flux.
Sources: AGENTS.md, specs/001-k8s-helm-backup/spec.md, specs/001-k8s-helm-backup/plan.md
Architectural Overview
The architecture implements a split-binary design built on a shared-core application controller. The infrahub-backup and infrahub-collect executables embed the KubernetesBackend driver to interface with cluster resources. Instead of compiled SDK libraries like client-go, the execution framework shells out directly to the kubectl CLI through a controlled CommandExecutor process layer.
The following diagram illustrates how the infrahub-ops-cli toolset communicates with Kubernetes workloads during operations:
The diagram shows the command flow starting from the CLI binaries, passing through the kubectl abstraction layer, and executing actions against targeted cluster pods.
Sources: AGENTS.md, specs/archive/003-collect-tool/plan.md
Workload Management and Helm Subcharts
A dedicated subchart named infrahub-backup coordinates Kubernetes deployments. This subchart works either as an integrated dependency of the main infrahub deployment or as an independent runtime workspace.
Sources: specs/001-k8s-helm-backup/plan.md
Subchart Workload Resources
The subchart manages three core operational workloads:
- CronJob Backup: Runs on a periodic schedule to trigger automated snapshots.
- One-Shot Job Backup: Executes a single, targeted backup ahead of planned maintenance windows.
- Restore Job: Recovers databases from existing storage objects.
Sources: specs/001-k8s-helm-backup/spec.md, specs/001-k8s-helm-backup/plan.md
The operational workflow for backup jobs maps directly to the active workload configurations:
The flowchart demonstrates the two ways to spawn a backup job and the subsequent data flows required to build and save the archive.
Sources: specs/001-k8s-helm-backup/spec.md, specs/001-k8s-helm-backup/plan.md, specs/archive/003-collect-tool/tasks.md
Workload Configuration Options
The behavior of these resources depends on specific Helm variables specified during deployment:
| Configuration Value | Type | Default Value | Description |
|---|---|---|---|
backup.enabled | Boolean | false | Enables backup orchestration. |
backup.mode | String | "job" | Sets the workload type to "job" or "cronjob". |
backup.schedule | String | "0 2 * * *" | Specifies the cron interval for periodic runs. |
backup.storage.type | String | "local" | Targets destination types: "s3" or "local". |
backup.storage.s3.secretName | String | "" | Names the secret containing AWS credentials. |
restore.enabled | Boolean | false | Enables cluster restore jobs. |
Sources: specs/001-k8s-helm-backup/plan.md
RBAC Role Declarations
Operational pods must have specific cluster access rights to track and interface with active application services. The subchart configures a dedicated ServiceAccount bound to a specific Role.
Sources: specs/001-k8s-helm-backup/plan.md, specs/001-k8s-helm-backup/spec.md
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: infrahub-backup
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "patch"]
Sources: [specs/001-k8s-helm-backup/plan.md()]
Troubleshooting Bundle Collection
The troubleshooting engine relies on the infrahub-collect binary to generate diagnostic support bundles. The collection framework runs outside the application context but inside the target cluster namespace to perform read-only discovery.
Sources: specs/archive/003-collect-tool/plan.md
Collection Process Flow
The collection execution proceeds through an ordered sequence of lifecycle phases:
The sequence diagram charts how the tool maps cluster resources, streams application container outputs, masks critical variables, and packages diagnostic data.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/tasks.md, specs/archive/003-collect-tool/opsmill-implement-report.md
Execution Safety Mechanisms
The framework implements specific constraints to guarantee safe environment execution:
- Lifecycle Isolation: The engine never restarts, modifies, or scales cluster pods during diagnostic checks.
- Time-Bounded Collection: Every subprocess runs under a controlled
exec.CommandContext. Log streams and file syncs time out after 5 minutes, while administrative status checks time out after 60 seconds. - Non-Fatal Resilience: Individual collector failures do not stop the execution loop. If a single service fails to respond, the orchestrator records a warning, writes the failure to the manifest, and continues processing the rest of the queue.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/tasks.md, specs/archive/003-collect-tool/opsmill-implement-report.md
Data Masking Engine
To protect sensitive operational data, all configuration dumps, process outputs, and environment variables pass through a central masking filter before hitting the storage layer. This engine performs a case-insensitive check for substrings matching password, secret, token, or key. It masks all identified target values with ********.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/tasks.md, specs/archive/003-collect-tool/opsmill-implement-report.md
Container Pruning and Retention Policies
When a backup job finishes successfully, the retention system triggers automated cleanup operations based on parameters specified by the operator.
The flowchart demonstrates the verification steps and safety loops executed before the retention rules remove any file from storage.
Sources: [specs/004-backup-retention-policy/plan.md()]
The pruning engine applies two independent filters to calculate the target removal list:
- Max Age: Identifies files older than a specified number of days.
- Max Count: Tracks total items and marks oldest entries for deletion when the count exceeds storage limits.
The retention engine evaluates rules independently for local directories and S3 buckets. An explicit safety floor overrides these calculations to ensure that at least the newest backup artifact remains in storage, preventing data loss during bulk pruning tasks.
Sources: [specs/004-backup-retention-policy/plan.md()]
Conclusion
Kubernetes Operations provide automated lifecycle capabilities through structured Helm subcharts and isolated, shell-out CLI utilities. This operational design isolates failures to protect running applications while providing automated retention schedules, real-time logging, data masking, and complete infrastructure recovery paths.
Sources: specs/001-k8s-helm-backup/spec.md, specs/archive/003-collect-tool/plan.md, specs/004-backup-retention-policy/plan.md