Content Discovery & Scoring
Relevant source files
The following files were used as context for generating this wiki page:
Content Discovery & Scoring is the core heuristic engine of the swe_distiller pipeline. Its primary purpose is to identify the most relevant "main" content of a web page while filtering out "noise" such as navigation menus, sidebars, headers, and footers. This system ensures that the resulting Markdown or JSON output is clean and readable, suitable for further processing by LLMs or human users.
The process relies on a combination of element-based scoring, structural noise analysis, and safety checks to validate that the extracted content is representative of the original document. It operates as a deterministic fallback when specific site extractors are unavailable or when the --llm mode is not utilized.
Sources: extensions/swe_distiller/ARCHITECTURE.md:16-24, extensions/swe_distiller/src/lib.rs:136-146
Content Discovery Logic
The discovery process follows a hierarchical selection strategy to find the best candidate for the main content. It prioritizes semantic tags like <article> before moving to common entry-point selectors and finally falling back to the <body> tag.
Discovery Flow
The following diagram illustrates how the system selects a content candidate:
The system first attempts to find the "best" article candidate by checking for <article> tags with at least 40 words. If none are found, it iterates through ENTRY_POINT_SELECTORS, combining their intrinsic score_element result with a priority value based on the selector's order.
Sources: extensions/swe_distiller/src/find_content.rs:13-44
Entry Point Selection
If no suitable article is found, the system evaluates nodes matching pre-defined selectors. The total score is calculated as:
Total Score = (Selector Priority) + (Intrinsic Element Score)
Priority is weighted based on the position in the ENTRY_POINT_SELECTORS list.
Sources: extensions/swe_distiller/src/find_content.rs:25-30
Scoring Mechanisms
The scoring system quantitatively distinguishes between high-value content and UI boilerplate. It uses word counts, tag density, and attribute indicators to calculate a weight for every candidate element.
Element Scoring (score_element)
The primary scoring function applies the following weights:
| Factor | Impact | Description |
|---|---|---|
| Word Count | +1.0 per word | Base score from plain text content. |
<p> tags | +10.0 per tag | Rewards dense paragraph structure. |
| Commas | +1.0 per comma | Heuristic for natural language complexity. |
| Image Density | -3.0 multiplier | Penalizes elements that are mostly images with little text. |
| Content Indicators | +15.0 flat | Bonus if class or id matches CONTENT_INDICATORS. |
| Link Density | Multiplier (1 - density) | Significant penalty for elements comprised mostly of links. |
Sources: extensions/swe_distiller/src/scoring.rs:16-43
Noise and Navigation Scoring
The system also evaluates elements likely to be non-content (e.g., sidebars) using score_non_content_block. It applies penalties (-10.0) for NAVIGATION_INDICATORS found in text and heavy penalties (-15.0) for high link density (> 50%).
Sources: extensions/swe_distiller/src/scoring.rs:45-62
Extraction Quality & Safety
After content is extracted or processed via the --llm pipeline, it undergoes safety checks to prevent hallucinations or massive data loss.
Safety Gates
The passes_safety_checks function compares the original heuristic extraction with a candidate (often from an LLM) to ensure structural integrity.
Structural Noise Scoring
A separate noise score is used to penalize "chrome" that might have leaked into the Markdown.
- Navigation keywords: Terms like "sign up", "search", or "share" add +3 to the noise score.
- Short lines: Lines with 2 or fewer words (that aren't headings) add +1.
- Repetitive runs: Non-empty runs of text exceeding 10 lines increase the score.
- Placeholders: Lone symbols like "!" or "·" add +2.
Sources: extensions/swe_distiller/src/extraction/checks.rs:1-35
Recovery and Refinement
If the initial extraction yields poor results (e.g., very low word count), the pipeline utilizes "Recovery Plans" to relax pruning constraints.
Recovery Pipeline
- Partial Selector Relaxation: If word count < 200, retries with
remove_partial_selectors = false. - Hidden Element Relaxation: If word count < 50, retries with
remove_hidden_elements = false. - Broad Relaxation: Final attempt disabling content patterns and low-scoring pruning if count remains < 50.
Sources: extensions/swe_distiller/src/lib.rs:244-275
Markdown Post-processing
The final content is refined using postprocess_markdown, which handles:
- Chrome Removal: Stripping known strings like "related articles" or "subscribe".
- Run-on Fixes: Separating headings or metadata that accidentally merged with body text.
- Bold Crossing: Fixing bold markers that span across heading boundaries.
Sources: extensions/swe_distiller/src/markdown.rs:168-185, extensions/swe_distiller/src/markdown.rs:238-265
Summary
Content Discovery and Scoring in swe_distiller provides a robust, heuristic-based foundation for web scraping. By combining prioritized DOM selection, complex scoring weights, and safety-focused validation, the system successfully distills raw HTML into high-quality text data while maintaining the flexibility to recover from overly aggressive pruning.
Sources: extensions/swe_distiller/ARCHITECTURE.md:16-24, extensions/swe_distiller/src/lib.rs:105-120