Log Collection Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The log collection strategy defines how the infrahub-collect tool gathers diagnostic information from an Infrahub deployment. It serves as a read-only troubleshooting mechanism that aggregates logs from multiple services and replicas into a unified support bundle. This strategy replaces legacy scripts with a Go-based implementation that supports both Docker Compose and Kubernetes environments.
Log collection prioritizes operational safety and data integrity by ensuring the process never mutates the workload lifecycle. The tool does not stop, start, or scale containers during the collection process. Each collector runs independently with defined time bounds, ensuring that a hung container in a degraded instance does not stall the entire collection run.
Sources: AGENTS.md, specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/spec.md:FR-010
Architecture and Core Primitives
The log collection system uses a shared-core architecture. A thin Cobra-based CLI entry point in src/cmd/infrahub-collect triggers orchestration logic located in src/internal/app. The strategy relies on a backend-agnostic collector framework that interacts with either Docker or Kubernetes through a narrow interface.
Logical Components
- Orchestrator: Manages the staging directory, executes the collector loop, and packages the final archive.
- Collectors: Environment-agnostic units of work that target specific services or diagnostics.
- Backend Seam: An interface providing primitives for replica enumeration and log streaming.
- Command Executor: Shells out to
dockerorkubectlCLIs usingexec.CommandContextto enforce timeouts.
Sources: AGENTS.md, specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/tasks.md:T009-T011
Data Flow for Log Streaming
The following diagram illustrates the flow from the user command to the final log file within the support bundle.
The diagram shows how the orchestrator triggers the log collector, which uses environment-specific backends to stream data into staging files before archiving.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/tasks.md:T011, T015, T030
Service and Replica Coverage
The strategy mandates the collection of logs for all Infrahub services present in the deployment. It captures logs from every replica of multi-instance services to ensure comprehensive diagnostic visibility.
Target Services
The system targets the following services for log collection:
infrahub-servertask-workerdatabase(Neo4j)message-queue(RabbitMQ)cache(Redis)task-managertask-manager-db(PostgreSQL)task-manager-background-svc
Sources: specs/archive/003-collect-tool/spec.md:FR-002, AGENTS.md
Environment Specifics
| Environment | Replica Identification | Previous Log Support |
|---|---|---|
| Docker Compose | Uses Container Name (e.g., <service>-<index>) | Not Supported |
| Kubernetes | Uses Pod Name and Container Name | Supported (captures logs from crashed/restarted pods) |
Sources: specs/archive/003-collect-tool/tasks.md:T015, T029, T030, specs/archive/003-collect-tool/spec.md:FR-002
Operational Constraints and Safety
Log collection is governed by strict operational rules to ensure it can function effectively on degraded systems without causing further issues.
Time-Bounded Execution
Every subprocess executed by a collector is time-bounded using context.WithTimeout. This prevents a single hung kubectl exec or docker exec command from blocking the entire troubleshooting run.
- Status Dumps/Exec: Default 60 seconds.
- Log Transfers/File Copies: Default 5 minutes.
Sources: specs/archive/003-collect-tool/tasks.md:T010, dev/guidelines/collectors.md
Failure Handling
Individual collector failures are treated as non-fatal. If a collector fails (e.g., due to a service being down), the system logs a warning, records the failure in the bundle manifest, and continues the run. The command ultimately exits with a code of 0, providing a partial bundle rather than no bundle at all.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/spec.md:FR-009, dev/guidelines/collectors.md
Data Volume Management
To prevent excessive disk usage, the system applies a per-service log volume limit.
- Default Limit: 100,000 lines.
- Configuration: Set via
--log-linesCLI flag orINFRAHUB_LOG_LINESenvironment variable. - Visibility: The effective limit is recorded in the bundle manifest.
Sources: specs/archive/003-collect-tool/plan.md, specs/archive/003-collect-tool/spec.md:FR-011
Privacy and Secret Masking
The log collection strategy includes a mandatory secret masking phase for environment and configuration dumps. The system uses a single choke-point (masking.go) to identify and redact sensitive information.
Masking Logic
The system performs case-insensitive substring matching on key names. At a minimum, any key containing the following terms is masked:
passwordsecrettokenkey
Redacted values are replaced with ********. This applies to environment variable dumps, Redis configuration, and RabbitMQ status outputs.
Sources: specs/archive/003-collect-tool/tasks.md:T005, specs/archive/003-collect-tool/spec.md:FR-008, dev/guidelines/collectors.md
Conclusion
The log collection strategy provides a robust, read-only diagnostic framework for Infrahub. By utilizing a split-binary architecture and environment-specific backends, the infrahub-collect tool ensures that troubleshooting data is gathered safely and consistently across Docker and Kubernetes deployments. The inclusion of mandatory timeouts, non-fatal failure handling, and automated secret masking makes the tool suitable for use on degraded production instances while protecting sensitive configuration data.