Logging & Observability

Logging and observability in swe_distiller are designed to provide structured insights into the extraction pipeline's internal operations. The system prioritizes debuggability, ensuring that failures are visible and diagnosis is cost-effective, adhering to the project's decision hierarchy which ranks observability above operational simplicity and maintainability. When the --debug flag is enabled, the system emits structured JSON logs to stderr, capturing critical lifecycle events, network acquisition metrics, and data transformation deltas.

The observability framework tracks the entire lifecycle of a URL distillation request, from initial HTTP acquisition through heuristic removal stages to optional LLM-based extraction. This allows developers and automated agents to monitor performance, verify extractor selection, and analyze the impact of various content cleanup passes on document size and word count.

Sources: AGENTS.md:28-30, extensions/swe_distiller/ARCHITECTURE.md:46-52, extensions/swe_distiller/src/observability.rs

Architecture and Data Flow

The observability logic is centralized in the observability module, which provides high-level functions for recording pipeline progress and transformation metrics. The system uses a structured approach where events are logged as JSON objects, facilitating easy parsing by external tools or automated agents.

Observability Event Flow

This diagram illustrates how observability calls are embedded within the core extraction pipeline.

The pipeline orchestrator (lib.rs) triggers specific logging events at each major transition point to ensure full visibility.

Sources: extensions/swe_distiller/src/lib.rs:25-66, extensions/swe_distiller/src/observability.rs

Core Observability Components

1. Structured Logging Functions

The system utilizes specific functions to handle different types of diagnostic information. These functions generally check the debug_enabled boolean before proceeding to write to stderr.

Sources: extensions/swe_distiller/src/observability.rs, extensions/swe_distiller/src/lib.rs:260-279

2. Event Categories

The pipeline tracks several distinct event categories to narrow down performance bottlenecks or extraction failures.

• Pipeline Events: Tracks pipeline.start, pipeline.extractor, and pipeline.done. It records total word counts and total parse_time_ms.
• Fetch Events: Tracks fetch.start, fetch.attempt, fetch.error, and fetch.http_success. It captures User-Agent strings, proxy usage (redacted), and HTTP status codes.
• Transformation Events: Recorded via apply_removal_step, these track the efficacy of selectors, hidden element removal, scoring-based pruning, and clutter-pattern removal.
• LLM Events: Tracks pipeline.llm outcomes and individual provider results (pipeline.llm.provider), including baseline noise scores and accepted noise scores.

Sources: extensions/swe_distiller/src/lib.rs:25-66, extensions/swe_distiller/src/fetch.rs:36-125, extensions/swe_distiller/src/extraction/llm.rs:27-58

Network & Fetch Observability

The fetch layer implements detailed logging for HTTP requests, including retries and fallbacks (like RSS or browser-based acquisition). To protect sensitive information, proxy credentials are automatically redacted before being logged.

Fetch Sequence with Observability

This sequence diagram shows the interaction between the fetch module and the observability layer during a typical request.

The proxy_log_value helper ensures that only the scheme, host, and port are recorded, preventing the leaking of user:pass credentials in debug logs.

Sources: extensions/swe_distiller/src/fetch.rs:36-125, extensions/swe_distiller/src/fetch.rs:334-348

Transformation and Removal Metrics

The distillation process involves multiple cleanup passes. The observability system monitors these passes by calculating the "delta" in content size for every step.

// extensions/swe_distiller/src/lib.rs:260-279
fn apply_removal_step<F>(debug_enabled: bool, step: &str, reason: &str, input: &str, f: F) -> String
where
    F: FnOnce(&str) -> String,
{
    let before_words = dom::count_words(&dom::strip_tags(input));
    let before_chars = input.chars().count();
    let output = f(input);
    let after_words = dom::count_words(&dom::strip_tags(&output));
    let after_chars = output.chars().count();
    observability::removal_step(
        debug_enabled,
        step,
        reason,
        before_words,
        after_words,
        before_chars,
        after_chars,
    );
    output
}

This wrapper function allows the system to log exactly how much "clutter" was removed by each specific logic block (e.g., selectors, hidden, scoring).

Sources: extensions/swe_distiller/src/lib.rs:260-279, extensions/swe_distiller/ARCHITECTURE.md:50

LLM Pipeline Observability

When the --llm mode is enabled, the system logs specific metrics regarding provider performance and safety checks.

• Baseline Noise: The noise score of the heuristic extraction is logged as a baseline.
• Provider Outcome: Records whether the LLM output was Applied or SkippedNoAcceptableCandidate.
• Safety Rejections: If an LLM candidate fails safety checks (such as word count drift or lack of heading overlap), the rejection is implicitly visible through the absence of an accepted_noise log for that provider.

Sources: extensions/swe_distiller/src/extraction/llm.rs:27-58, extensions/swe_distiller/src/extraction/llm.rs:62-68

Roadmap and Future Improvements

The TODO.md highlights planned enhancements for the observability module:

• Stable Event Schema: Implementing a formal schema for all debug logs to ensure long-term compatibility with log aggregators.
• Summary Diagnostics: Adding an optional summary output specifically tailored for automated agents to quickly assess extraction quality.
• Provider Metrics: Expanding LLM logs to include specific latency per provider and explicit error categories (e.g., timeout, bad response).

Sources: extensions/swe_distiller/TODO.md:14-16, extensions/swe_distiller/TODO.md:29-31

Logging and observability are integral to the swe_distiller pipeline, providing the necessary transparency to debug complex HTML extraction and ensure high-quality Markdown output. By capturing structured data at every stage—from network fetch to final LLM refinement—the system maintains high levels of debuggability and operational visibility.