Diagnostics & Metrics
Relevant source files
The following files were used as context for generating this wiki page:
Diagnostics and metrics collection within the Infrahub ecosystem provides a read-only troubleshooting framework for gathering system state, performance data, and service logs. The system aggregates information from databases, message queues, caches, and application workers into a unified troubleshooting bundle to support debugging and infrastructure analysis.
The core of this functionality resides in the infrahub-collect tool, which implements an environment-agnostic collector framework. This framework executes a sequence of specialized collectors that target specific infrastructure components while ensuring zero mutation of the running workload. Individual collector failures are treated as non-fatal, allowing the tool to produce a partial bundle even if some services are unreachable.
Sources: AGENTS.md, specs/archive/003-collect-tool/plan.md
Collection Framework Architecture
The collection process follows an orchestrated loop that stages data in a temporary directory before archiving it into a .tar.gz bundle. An orchestrator manages the execution flow, handling timeouts and recording the outcome of each individual collector in a central manifest file.
The diagram shows the high-level workflow of the collection orchestrator, from staging directory creation to final archive generation.
Sources: src/internal/app/collect.go, specs/archive/003-collect-tool/plan.md
Key Components
| Component | Description | Source |
|---|---|---|
CollectBundle | Main orchestrator function managing the staging/archive lifecycle. | src/internal/app/collect.go |
bundle_information.json | A manifest file containing metadata and per-collector execution results. | src/internal/app/collect_manifest.go |
collectBackend | Interface defining environment-specific primitives for replicas, logs, and metrics. | src/internal/app/collect.go |
safeRun / safeSkip | Wrappers that handle collector panics and record failures in the manifest. | specs/archive/003-collect-tool/opsmill-implement-report.md |
Diagnostic Collectors
Diagnostic collectors gather stateful information and configuration from infrastructure services. These collectors utilize command execution (e.g., exec.CommandContext) to interact with services via their native CLI tools.
Sources: src/internal/app/collect_diagnostics.go, specs/archive/003-collect-tool/plan.md
Service-Specific Diagnostics
The system implements specialized collection logic for the following services:
- Database (Neo4j): Copies
neo4j.loganddebug.log. If requested via--include-queries, it copies the full log directory including query logs. - Message Queue (RabbitMQ): Executes
rabbitmqctlcommands to list queues, exchanges, bindings, and connections. Outputs are masked to protect secrets. - Cache (Redis): Executes
redis-clicommands forinfo,client list,slowlog get, anddbsize. - Task Manager: Gathers data on work pools, work queues, flow runs, and events using the Prefect CLI or embedded Python scripts.
- Server Info: Retrieves the Infrahub version, Python package list (
pip list), and environment variables (masked).
Sources: src/internal/app/collect_diagnostics.go:1-150, specs/archive/003-collect-tool/tasks.md:T018-T023
Masking and Redaction
To protect sensitive information, diagnostic collectors route specific outputs through a masking choke-point. The system performs case-insensitive substring matching on keys containing password, secret, token, or key and replaces their values with ********.
Sources: src/internal/app/masking.go, specs/archive/003-collect-tool/plan.md
Metric Collection
Metric collectors gather resource utilization data from the deployment environment. The implementation differs based on whether the environment is Docker Compose or Kubernetes.
Metric Sources
- Docker Backend: Executes
docker stats --no-streamacross all containers in the project. - Kubernetes Backend: Executes
kubectl top podswithin the configured namespace. If themetrics-serveris missing, the collector records a failure in the manifest but does not stop the bundle generation.
Sources: src/internal/app/collect_metrics.go, specs/archive/003-collect-tool/tasks.md:T016, T031
The sequence diagram illustrates how metric collection requests flow from the orchestrator through backend-specific CLI commands.
Sources: src/internal/app/collect_metrics.go, src/internal/app/collect.go
Manifest and Metadata
Every bundle contains a bundle_information.json file. This manifest provides the source of truth regarding what was collected and the status of each operation.
Manifest Data Model
| Field | Description |
|---|---|
collect_id | Unique identifier for the collection run. |
infrahub_version | The version of Infrahub detected during collection. |
environment | The detected environment (e.g., docker or kubernetes). |
status | The outcome of a collector: success, failed, skipped, or timed out. |
reason | A descriptive error message required if a collector is not successful. |
Sources: src/internal/app/collect_manifest.go, specs/archive/003-collect-tool/tasks.md:T008
Operational Constraints
Diagnostics and metrics collection must adhere to strict operational safety principles to ensure that troubleshooting does not exacerbate existing issues.
- Read-Only Operations: The collection process never mutates the workload lifecycle. It does not stop, start, or scale containers.
- Time-Bounded Execution: Every collector subprocess is time-bounded using
context.Context. Default timeouts are 60 seconds for execution dumps and 5 minutes for log or file transfers. - Offline by Default: The tool does not pull images or require network egress beyond the target deployment during standard collection.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/critiques/critique-20260703-170700.md:E1
Summary
The Diagnostics & Metrics system provides a robust, non-intrusive mechanism for capturing the state of an Infrahub deployment. By utilizing an environment-agnostic collector framework with built-in masking, timeouts, and manifest tracking, the project ensures that operators can gather critical troubleshooting data safely across both Docker and Kubernetes platforms.