Metadata Extraction
Relevant source files
The following files were used as context for generating this wiki page:
Metadata Extraction in swe_distiller is a multi-layered process designed to identify and preserve key document attributes such as titles, authors, publication dates, and site information. This process runs as a integral part of the content distillation pipeline, combining deterministic heuristic searches with structured data parsing and optional LLM-based overrides. Sources: extensions/swe_distiller/ARCHITECTURE.md, extensions/swe_distiller/src/lib.rs:245-285
The extraction engine prioritizes high-fidelity sources like Schema.org (JSON-LD) and OpenGraph tags, falling back to document heuristics (e.g., finding bylines in the first 20 lines of Markdown) when structured data is absent. Sources: extensions/swe_distiller/src/metadata.rs:77-130, extensions/swe_distiller/src/metadata.rs:219-232
Extraction Architecture and Flow
The metadata extraction logic is orchestrated within the parse_internal function. It follows a hierarchy of priority: site-specific extractor overrides, followed by structured HTML metadata, and finally heuristic analysis of the generated Markdown content. Sources: extensions/swe_distiller/src/lib.rs:188-250
The diagram above illustrates the fallback mechanism used to populate the metadata fields in the final response object.
Sources: extensions/swe_distiller/src/lib.rs:245-275, extensions/swe_distiller/src/metadata.rs
Core Extraction Components
1. Structured Data Parsing
The system actively searches for application/ld+json script tags to extract Schema.org data. This data is converted into a serde_json::Value and queried using path-based lookups (e.g., publisher.name, author.name). Sources: extensions/swe_distiller/src/metadata.rs:31-60
2. Meta Tag Collection
The collect_meta_tags function iterates through all <meta> tags, capturing name, property, and content attributes into MetaTag structures. Sources: extensions/swe_distiller/src/metadata.rs:18-28
3. Heuristic Fallbacks
When structured data fails, the system employs several heuristic strategies:
- Title Cleaning: Strips common site suffixes (e.g., " | Site Name") using delimiters like
|,-, or—. Sources: extensions/swe_distiller/src/metadata.rs:172-184 - Byline Detection: Searches the first 20 lines of the Markdown output for lines starting with "By " or "by ". Sources: extensions/swe_distiller/src/metadata.rs:219-232
- Date Regex: Uses a complex regular expression to find date patterns (ISO format or Month-Day-Year) within the text. Sources: extensions/swe_distiller/src/metadata.rs:8-15
Metadata Fields and Priority
The following table summarizes the data sources and search keys for standard metadata fields.
| Field | Primary Source (Schema Paths) | Secondary Source (Meta Keys) | Heuristic Fallback |
|---|---|---|---|
| Title | headline, name | og:title, twitter:title, dc.title | First <h1> or <title> tag |
| Author | author.name, creator.name | author, article:author, parsely-author | "By [Name]" in text |
| Published | datePublished, dateCreated | article:published_time, date | <time> tag or Regex match |
| Site | publisher.name, isPartOf.name | og:site_name, application-name | Domain from URL |
| Description | description | og:description, twitter:description | First 3 lines of content |
Sources: extensions/swe_distiller/src/metadata.rs:62-126, extensions/swe_distiller/src/lib.rs:252-273
Metadata Block Removal
To prevent metadata from appearing twice (once in the extracted fields and once in the article body), the pipeline includes a dedicated removal step. The remove_metadata_block function identifies and deletes HTML blocks (divs, paragraphs, or sections) that immediately follow an <h1> tag and contain date patterns, provided that metadata has already been successfully extracted. Sources: extensions/swe_distiller/src/removal/metadata_block.rs:24-60
Sequence showing the relationship between extraction and content cleanup.
Sources: extensions/swe_distiller/src/lib.rs:230-239, extensions/swe_distiller/src/removal/metadata_block.rs
LLM-Enhanced Metadata Extraction
When the --llm mode is enabled, the pipeline can override heuristic metadata. The apply_llm_markdown_override function takes the LLM-generated Markdown and re-extracts the title and description to ensure the structured response matches the refined content. Sources: extensions/swe_distiller/src/lib.rs:327-343, extensions/swe_distiller/src/extraction/llm.rs
LLM Safety Checks
Before accepting an LLM-extracted metadata or content candidate, the system performs safety checks, including comparing the first heading of the original vs. the candidate. If the heading token overlap is less than 50%, the LLM candidate is rejected to prevent "hallucinated" titles. Sources: extensions/swe_distiller/src/extraction/checks.rs:48-52, extensions/swe_distiller/src/extraction/checks.rs:69-91
Conclusion
Metadata extraction in swe_distiller is designed to be robust against varying website structures. By layering structured JSON-LD parsing over traditional Meta tag extraction and supplementing with text-based heuristics and LLM validation, the system ensures that critical document context is preserved even when the primary content is heavily stripped or transformed. Sources: extensions/swe_distiller/src/lib.rs:188-285, extensions/swe_distiller/ARCHITECTURE.md