Observability & Logging

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• dev/constitution.md
• README.md
• dev/guidelines/writing-an-adapter.md
• tasks/bench.py
• pyproject.toml

Observability in infrahub-sync provides visibility into the data synchronization process through structured logging and performance benchmarking. The system uses structured logs to track sync progress, object counts, and connection events, enabling integration with external observability stacks like Splunk, Datadog, or Loki.

Logging serves as a primary tool for troubleshooting and monitoring sync health. It records high-level operations such as loading data, calculating differences, and applying changes to destination systems. The project enforces strict security boundaries by prohibiting the inclusion of credentials or secrets within any log output or traceback.

Sources: AGENTS.md, README.md:65-67, dev/constitution.md:124-127

Structured Logging Architecture

The project implements logging via the structlog library. Unlike standard print statements, structured logging allows for programmatic parsing and filtering of log data by including key-value context.

Implementation Principles

• Structured Output: Use structlog for all production code; avoid print() except for simple CLI diagnostics or benchmark scripts.
• Contextual Information: Logs include relevant metadata such as request IDs, API endpoints, object counts, and specific model types.
• Security Constraints: Logs MUST NOT contain credentials, passwords, tokens, or full authorization headers. This applies to both standard logs and exception tracebacks.
• Adapter Identification: Adapters define a type class attribute used to identify the actor in log lines.

Sources: AGENTS.md, dev/constitution.md:124-127, dev/guidelines/writing-an-adapter.md:46-55

Log Generation Flow

The following diagram illustrates how different components contribute context to the structured logging system:

The logging flow centralizes data from the engine, adapters, and CLI into a single structured output stream.
Sources: README.md:65-67, dev/guidelines/writing-an-adapter.md:46-55, dev/constitution.md:124-127

Sync Phase Observability

The Potenda engine and benchmarking tasks track time and progress across distinct phases of the synchronization lifecycle.

Logged Sync Phases

Phase	Description	Key Observability Metrics
Load	Importing data from source and destination systems.	Endpoint access, object counts per model.
Diff	Comparing source data against destination data.	Number of changes detected (Create/Update/Delete).
Sync	Applying changes to the destination system.	Wall-clock time per tier, completion status.

Sources: tasks/bench.py:192-211, dev/constitution.md:128-130

Benchmarking and Performance Tracking

The bench.py task provides specialized observability for performance analysis. It measures wall-clock time for different sync scenarios, including "cold" runs (empty cache) and "warm" runs (utilizing existing cache/destination state).

# From tasks/bench.py:192-205
def _parse_phase_timings(stdout: str) -> tuple[float | None, float | None, list[str]]:
    """Best-effort phase splits from structlog output."""
    notes: list[str] = []
    load_count = len(_LOAD_LINE_RE.findall(stdout))
    sync_done = _SYNC_DONE_RE.search(stdout)
    sync_seconds = float(sync_done.group(1)) if sync_done else None

    tier_count = len(_SYNC_TIER_RE.findall(stdout))
    if tier_count:
        notes.append(f"{tier_count} tiers logged")

Sources: tasks/bench.py:192-205

Error Handling and Exception Logging

Observability includes clear reporting of failures to maintain the "Sync Idempotency & Safety" principle. The system prioritizes explicit error handling over silent failures.

Error Reporting Requirements

• Specific Exceptions: Adapters must raise narrow, specific exceptions (e.g., authentication failures 401/403, timeouts) rather than catching general exceptions.
• Partial Failures: The system must clearly surface when only a subset of records fails to sync, preventing ambiguous destination states.
• Traceback Safety: Ensure that automated error reports and tracebacks do not leak internal configuration details or secrets.

Sources: dev/constitution.md:28-36, dev/guidelines/writing-an-adapter.md:36-44

Summary of Observability Tooling

Component	Tool / Library	Purpose
Logging	structlog	Structured, contextual system logs.
CLI Feedback	tqdm	Visual progress bars for long-running syncs.
Performance	tasks/bench.py	Comparative wall-clock timing for sync scenarios.
Validation	ruff/pylint/ty	Static analysis to ensure logging standards are met.

Sources: pyproject.toml:23, 26, tasks/bench.py:13-17, dev/constitution.md:124-127

Observability and logging in infrahub-sync ensure that every synchronization event is traceable and measurable. By combining structured logs with performance benchmarking, operators can maintain visibility into their infrastructure state while adhering to strict security and idempotency requirements.