DOM Standardization
Relevant source files
The following files were used as context for generating this wiki page:
DOM Standardization is a critical structural normalization phase within the swe_distiller pipeline. Its primary purpose is to clean and format extracted HTML content into a predictable, high-quality state before it is converted into Markdown or JSON. This stage follows content selection and initial cleanup, ensuring that specific elements like headings, footnotes, and callouts conform to a standard structure while removing noise such as empty tags and excessive whitespace.
The standardization process is deterministic and occurs as part of the core extraction pipeline. It prepares the document for the final output contract, which requires a clean DistillerResponse containing the standardized content and metadata.
Sources: extensions/swe_distiller/ARCHITECTURE.md:4-16, extensions/swe_distiller/src/lib.rs:136-184
Pipeline Integration
DOM Standardization is integrated into the parse_internal function, where it acts as a series of transformation passes on the content_html string. These passes occur after site-specific extraction and before final Markdown conversion.
The diagram shows the sequential placement of standardization steps within the swe_distiller extraction pipeline.
Sources: extensions/swe_distiller/src/lib.rs:132-184, extensions/swe_distiller/ARCHITECTURE.md:21-31
Content Normalization Logic
The standardize_content module applies a suite of regular expressions to the HTML string to enforce structural consistency. This includes collapsing non-breaking spaces, removing HTML comments, and downgrading H1 tags to H2 to ensure the document has a logical hierarchy (reserving H1 for the page title).
Key Transformations
| Transformation | Implementation Detail | Purpose |
|---|---|---|
| Heading Downgrade | Replaces <h1> with <h2> | Ensures document hierarchy consistency. |
| Empty Tag Removal | Removes empty <p>, <div>, <section>, <article>, <span> | Eliminates structural noise. |
| Whitespace Collapse | Reduces [ \t]{2,} to a single space | Normalizes inline spacing. |
| Line Break Capping | Reduces <br> sequences of 3+ to exactly two | Prevents excessive vertical whitespace. |
| Entity Normalization | Replaces with standard space | Improves text readability. |
Sources: extensions/swe_distiller/src/standardize/content.rs:19-33
Implementation Example
pub fn standardize_content(input: &str, _page_title: &str) -> String {
let mut out = input.to_string();
out = NBSP_RE.replace_all(&out, " ").to_string();
out = COMMENT_RE.replace_all(&out, "").to_string();
out = H1_RE.replace_all(&out, "<h2$1>").to_string();
out = out.replace("</h1>", "</h2>");
out = EMPTY_P_RE.replace_all(&out, "").to_string();
// ... further removals ...
out = EXCESS_BR_RE.replace_all(&out, "<br><br>").to_string();
out = WS_RE.replace_all(&out, " ").to_string();
out.trim().to_string()
}
Sources: extensions/swe_distiller/src/standardize/content.rs:20-32
Metadata and Title Standardization
Standardization also extends to the final Markdown representation. The pipeline ensures that a title heading is present and correctly formatted. If the article body lacks an H1, the ensure_title_heading function injects one based on extracted metadata (e.g., og:title or <title> tag).
Title Injection Logic
- Check for H1: If the markdown already contains a
#heading, no action is taken. - Heuristic Check: If the title text already appears early in the document (first 400 characters), injection is skipped to avoid redundancy.
- Injection: Otherwise, a
# {title}heading is prepended to the content.
Sources: extensions/swe_distiller/src/markdown.rs:135-159, extensions/swe_distiller/src/lib.rs:188-193
Markdown Post-processing
Following the structural HTML standardization and conversion, a post-processing phase is applied to the Markdown to fix common conversion artifacts.
The diagram illustrates the sequential post-processing steps applied to the generated Markdown string.
Sources: extensions/swe_distiller/src/markdown.rs:161-170
Bold Span Correction
The fix_bold_spans_crossing_headings function is a specific standardization pass that prevents bold markdown markers (**) from spanning across heading boundaries, which often occurs when HTML <strong> tags wrap block-level headings. It detects unmatched bold markers and closes them before the heading starts.
Sources: extensions/swe_distiller/src/markdown.rs:252-290
Conclusion
DOM Standardization in swe_distiller transforms heterogeneous web content into a clean, structurally sound intermediate representation. By utilizing targeted regex-based passes for content normalization and specific structural fixes during Markdown conversion, the system ensures that the final output is free of common web artifacts like navigation chrome, empty elements, and inconsistent heading hierarchies.
Sources: extensions/swe_distiller/ARCHITECTURE.md:45-51, extensions/swe_distiller/src/lib.rs:136-224