Removal & Cleanup Stages

Relevant source files

The following files were used as context for generating this wiki page:

• extensions/swe_distiller/src/removal/content_patterns.rs
• extensions/swe_distiller/src/removal/metadata_block.rs
• extensions/swe_distiller/src/removal/selectors.rs
• extensions/swe_distiller/src/lib.rs
• extensions/swe_distiller/src/markdown.rs
• extensions/swe_distiller/ARCHITECTURE.md
• extensions/swe_distiller/src/observability.rs

Introduction

The Removal & Cleanup Stages form a critical phase in the swe_distiller pipeline, responsible for transforming raw HTML into a clean, readable format suitable for Markdown conversion. This stage follows content acquisition and initial parsing, applying a series of deterministic heuristic passes to prune non-essential elements such as navigation bars, footers, social media widgets, and repetitive UI chrome.

The system is designed to be fast and observable, tracking deltas in word and character counts at every step to ensure transparency in the distillation process. These stages are primarily orchestrated in the parse_internal function, which sequentially applies different removal modules based on the provided configuration options.
Sources: extensions/swe_distiller/ARCHITECTURE.md:14-23, extensions/swe_distiller/src/lib.rs:136-200

Pipeline Orchestration

The cleanup process is structured as a series of sequential passes. Each pass is wrapped in an apply_removal_step utility that handles telemetry, logging the state of the document before and after the transformation.

The diagram above illustrates the sequential flow of content through various cleanup modules as implemented in the core library.
Sources: extensions/swe_distiller/src/lib.rs:136-190, extensions/swe_distiller/src/observability.rs:20-40

Removal Stage Configuration

The intensity and scope of the removal stages are controlled via the DistillerOptions struct. Users can toggle specific cleanup heuristics to balance between aggressive noise reduction and content preservation.

Option	Description	File Reference
remove_exact_selectors	Removes specific tags like nav, footer, and header.	src/removal/selectors.rs
remove_partial_selectors	Uses regex to find and remove classes/IDs containing clutter keywords.	src/removal/selectors.rs
remove_hidden_elements	Prunes elements with display: none or hidden attributes.	src/lib.rs:163
remove_content_patterns	Removes text-based boilerplate (e.g., "5 min read", newsletter signups).	src/removal/content_patterns.rs
remove_low_scoring	Drops blocks identified as low-value by heuristic scoring.	src/lib.rs:182

Sources: extensions/swe_distiller/src/lib.rs:150-190, extensions/swe_distiller/src/removal/selectors.rs:9-40

Detailed Removal Modules

Selector-Based Pruning

This module targets specific HTML structures. It consists of two sub-modes:

1. Exact Selectors: Removes standard non-content tags such as <nav>, <aside>, <footer>, and <header>. It also handles script and style tag removal via specialized regex.
2. Partial Selectors: Iterates through PARTIAL_PATTERNS to find elements where class or ID attributes contain substrings associated with chrome (e.g., "sidebar", "menu").

Sources: extensions/swe_distiller/src/removal/selectors.rs:9-40

Content Pattern Pruning

The content_patterns module uses pre-compiled regular expressions to target recurring UI boilerplate and non-article text.

• Read Time: Removes strings matching \b\d+\s*min(?:ute)?s?\s+read\b.
• Newsletter/Subscriptions: Matches div or p tags containing "subscribe" or "newsletter" keywords.
• Social/Chrome Blocks: Targets small blocks containing "Sign up", "Follow", "Share this article", or standalone counters.
• Truncation: If a "Related Content" heading (e.g., "You might also like") is detected, the document is truncated at that point to prevent leaking of bottom-of-page clutter.

Sources: extensions/swe_distiller/src/removal/content_patterns.rs:4-53

Metadata Block Removal

This specialized stage targets the "byline" information that often appears immediately after the main <h1> title. If the system has already successfully extracted the author or publication date into the document's metadata, it removes the redundant HTML block containing this information to keep the Markdown body focused on the article content.
Sources: extensions/swe_distiller/src/removal/metadata_block.rs:24-60

Markdown Post-Processing

After the HTML has been cleaned and converted to Markdown, a final post-processing pass is applied to ensure structural integrity and further remove "chrome" that might have survived the HTML phase.

Post-Processing Steps

1. Metadata Run-on Normalization: Fixes cases where links and dates are mashed together without proper spacing.
2. Chrome Stripping: Filters lines that match known action phrases (e.g., "Listen", "Write a response") and removes standalone digits or placeholder characters like "!" or "·".
3. Bold Span Correction: Fixes Markdown syntax errors where bold markers (**) cross heading boundaries, ensuring headers remain valid.

pub fn postprocess_markdown(input: &str) -> String {
    let mut out = html_entity_decode(input);
    out = normalize_metadata_runons(&out);
    out = strip_known_chrome(&out);
    out = fix_bold_spans_crossing_headings(&out);
    out = MULTI_NEWLINE_RE.replace_all(&out, "\n\n").to_string();
    out.trim().to_string()
}

Sources: extensions/swe_distiller/src/markdown.rs:163-170, extensions/swe_distiller/src/markdown.rs:219-270

Recovery and Relaxation

Because aggressive removal can sometimes delete actual content, the pipeline includes a Recovery Plan mechanism. If the word count of the resulting document falls below specific thresholds (e.g., 200 words), the pipeline automatically re-runs with relaxed removal options (e.g., disabling remove_partial_selectors) to see if content can be recovered.
Sources: extensions/swe_distiller/src/lib.rs:94-110, extensions/swe_distiller/src/lib.rs:260-290

Summary

The Removal & Cleanup Stages provide a multi-layered defense against web clutter. By combining tag-based pruning, regex-driven pattern matching, and post-Markdown sanitization, swe_distiller produces high-quality, readable documents while maintaining high performance through deterministic heuristics. The inclusion of recovery plans ensures that the pipeline remains robust even when faced with aggressive anti-clutter settings.