Troubleshooting & Logging
Relevant source files
The following files were used as context for generating this wiki page:
Troubleshooting and logging within Harbor Satellite provide critical visibility into the health and operational status of both the Satellite edge component and the Ground Control management service. The system utilizes structured logging (JSON) and proactive health checking to ensure that connectivity between edge nodes, the central management layer, and the upstream Harbor registry remains stable.
The logging infrastructure is built on zerolog, offering configurable verbosity levels and structured output for automated log analysis. Troubleshooting is further supported by automated validation of configurations and health check probes that monitor upstream dependencies like Harbor core services and local OCI registries.
Logging Infrastructure
The Harbor Satellite ecosystem uses the zerolog library to provide structured, high-performance logging. By default, the system supports both human-readable and JSON-formatted logs, which can be toggled via CLI flags or configuration files.
Configuration and Verbosity
Log levels are strictly enforced during configuration validation. If an invalid log level is provided, the system defaults to info to ensure continuous visibility while alerting the user.
| Log Level | Description |
|---|---|
debug | Highly verbose output for development and deep troubleshooting |
info | Default level; captures standard operational milestones |
warn | Non-fatal issues, such as configuration defaults being applied |
error | Fatal operations or failed sync processes |
panic/fatal | Critical failures causing component shutdown |
Sources: pkg/config/validate.go:76-88, cmd/main.go:48, CLAUDE.md:124-128
Logging Flow and Initialization
The Satellite component initializes its logger early in the execution lifecycle, incorporating warnings generated during the initial configuration scan.
The initialization process ensures that any issues found during the validation of state_replication_interval or ground_control_url are immediately visible in the logs before the core services start.
Sources: cmd/main.go:136-149, internal/utils/utils.go:142-152
Health Monitoring
Harbor Satellite implements health checks at multiple levels to facilitate rapid troubleshooting of connectivity and dependency issues.
Harbor Health Checks
Ground Control performs proactive health checks on the upstream Harbor registry. It verifies that core components are operational before allowing management tasks to proceed.
| Component | Status | Action |
|---|---|---|
| Core API | Mandatory | Must return 200 OK from /api/v2.0/health |
| Portal | Optional | Skipped by default in health check |
| Trivy | Optional | Skipped by default in health check |
| Jobservice | Optional | Skipped by default in health check |
Sources: ground-control/internal/harborhealth/check.go:19-29, ground-control/internal/harborhealth/check.go:50-70
SPIFFE/SPIRE Troubleshooting
In production environments utilizing SPIFFE/SPIRE for identity, specific troubleshooting steps are required to verify the trust chain.
Common Troubleshooting Scenarios
Configuration Validation Errors
The system performs strict validation on file paths and network addresses. Common issues include:
- TLS Files: Both
cert_fileandkey_filemust exist if one is provided. - Registry Addresses: IPv6 addresses are currently unsupported; the system strictly validates for IPv4.
- Cron Expressions: Invalid schedules for
state_replication_intervalorheartbeat_intervalare automatically reset to system defaults with a warning log.
Sources: pkg/config/validate.go:174-205, internal/utils/utils.go:21-31, pkg/config/validate.go:117-133
Connectivity and Replication
If the Satellite fails to replicate artifacts, the following diagnostic checklist is derived from the service logic:
- Check Registration: Verify the Satellite is registered with Ground Control (returns a token).
- Verify Ground Control URL: Ensure
ground_control_urlis reachable viacurl. - Local Registry (Zot): Ensure the local registry is running on the configured port (default 8585).
- CRI Mirroring: Check if the container runtime (containerd/docker) has been updated with the local mirror config via the
--mirrorsflag.
Sources: deploy/no-spiffe/quickstart.md:144-165, cmd/main.go:94-105
Diagnostic Utilities
The repository provides several scripts and tools for testing and validating the logging/troubleshooting flow:
- E2E Tests: Use
task e2eto run full integration tests which output logs to/tmp/gc.logand/tmp/sat.log. - Config Watcher: The system includes a file watcher that triggers a "Hot Reload" on configuration changes, logging specific "changes" and "warnings" without requiring a restart.
- PrintData: A utility function in the codebase allows for printing file contents with line numbers to the console for manual inspection of configuration files.
Sources: scripts/e2e-test.sh:65-75, cmd/main.go:167-194, internal/utils/utils.go:108-115
Troubleshooting Harbor Satellite relies on a combination of structured logs from zerolog, dependency health checks against Harbor, and strict configuration validation. By monitoring the logs for warn and error levels, administrators can identify issues with TLS certificates, network connectivity, or invalid cron schedules before they impact artifact distribution.