Troubleshooting Bundles
Relevant source files
The following files were used as context for generating this wiki page:
Troubleshooting bundles are diagnostic archives collected from Infrahub deployments to assist support teams in identifying and resolving issues. The infrahub-collect tool gathers per-service logs, diagnostic status dumps, and container metrics into a single compressed package. This tool operates as a read-only utility, ensuring that collection never mutates or interrupts the workload lifecycle of the Infrahub instance.
Sources: AGENTS.md:12, specs/archive/003-collect-tool/plan.md:10-15, specs/archive/003-collect-tool/spec.md:126-128
System Architecture
The Troubleshooting Bundle system follows a split-binary architecture, where a specialized CLI (infrahub-collect) interacts with a shared internal application core. The orchestrator manages the collection lifecycle, executing individual collectors within bounded timeouts.
Collection Workflow
The orchestrator initializes a staging directory, iterates through a planned sequence of collectors, and finalizes the run by creating a compressed archive.
This diagram illustrates the sequential execution of collectors and the subsequent packaging of diagnostic data.
Sources: specs/archive/003-collect-tool/plan.md:11-16, src/internal/app/collect.go:88-106
Core Components
The Orchestrator
The CollectBundle function in src/internal/app/collect.go acts as the central orchestrator. It manages the lifecycle of the collection process, ensuring that individual collector failures are non-fatal. If a collector fails or times out, the orchestrator logs a warning, records the failure in the manifest, and continues to the next collector.
Sources: src/internal/app/collect.go:88, specs/archive/003-collect-tool/plan.md:38-40
Collector Framework
Collectors are units of work that gather specific diagnostic data. The framework supports the following categories:
- Service Logs: Retrieves logs for all replicas, including previous container logs on Kubernetes.
- Diagnostics: Executes commands within containers to dump status for Neo4j, RabbitMQ, Redis, and Prefect.
- Metrics: Captures container resource utilization (e.g.,
docker statsorkubectl top). - Extras: Opt-in data such as backups (
--include-backup) or benchmarks (--benchmark).
Sources: AGENTS.md:85-94, specs/archive/003-collect-tool/plan.md:10-16
Data Security and Masking
To protect sensitive information, the system implements key-name secret masking. The masking.go utility performs case-insensitive substring matches for keywords like password, secret, token, and key. Values associated with these keys in environment variable dumps or configuration files are replaced with ********.
Sources: specs/archive/003-collect-tool/plan.md:16, specs/archive/003-collect-tool/spec.md:118-120
Archive Structure and Manifest
The final output is a support_bundle_<timestamp>.tar.gz file. Every bundle contains a bundle_information.json manifest that serves as the authoritative record of the collection run.
Bundle Layout
| Directory | Content |
|---|---|
logs/ | Per-service replica logs (e.g., <pod>.log, <pod>_<container>.log). |
database/ | Neo4j server logs (neo4j.log, debug.log). |
message-queue/ | RabbitMQ status, queues, and environment dumps. |
cache/ | Redis info, client list, and masked configuration. |
task-manager/ | Prefect work pools, flow runs, and events. |
server/ | Version info, package lists, and masked environment variables. |
metrics/ | Resource usage data from the environment backend. |
Sources: specs/archive/003-collect-tool/contracts/bundle-layout.md:7-30
Manifest Schema
The BundleManifest struct tracks technical metadata and collector results.
type BundleManifest struct {
ManifestVersion int `json:"manifest_version"`
CollectID string `json:"collect_id"`
Timestamp string `json:"timestamp"`
InfrahubVersion string `json:"infrahub_version"`
Environment string `json:"environment"`
LogLines int `json:"log_lines"`
Collectors []CollectorResult `json:"collectors"`
}
Sources: src/internal/app/collect_manifest.go:33-41
CLI Configuration
The infrahub-collect command supports several flags to control the scope and output of the troubleshooting bundle.
| Flag | Environment Variable | Default | Description |
|---|---|---|---|
--output-dir | INFRAHUB_OUTPUT_DIR | ./infrahub_bundles | Directory to save the final archive. |
--log-lines | INFRAHUB_LOG_LINES | 100000 | Number of lines to tail from each log. |
--include-backup | INFRAHUB_INCLUDE_BACKUP | false | Includes a full backup next to the bundle. |
--include-queries | INFRAHUB_INCLUDE_QUERIES | false | Includes Neo4j query logs. |
--benchmark | INFRAHUB_BENCHMARK | false | Runs host-performance benchmarks. |
Sources: src/cmd/infrahub-collect/main.go:50-70, specs/archive/003-collect-tool/plan.md:28, specs/archive/003-collect-tool/spec.md:130-132
Operational Constraints
Collection is designed for safety in degraded environments:
- Read-Only: The tool never stops, restarts, or scales workloads.
- Time-Bounded: Individual collectors use
exec.CommandContextwith timeouts (typically 60s for status dumps, 5m for transfers) to prevent hung containers from stalling the entire process. - Offline-Ready: The tool performs no network egress and assumes no internet access, except when explicitly pulling a benchmark image.
Sources: specs/archive/003-collect-tool/plan.md:38-42, specs/archive/003-collect-tool/spec.md:126-128
The Troubleshooting Bundle system provides a robust, non-intrusive method for gathering critical diagnostic data across Docker and Kubernetes environments, centralizing logs and system state into a standardized, masked archive.