The Extractor Registry

The Extractor Registry is a core component of the swe_distiller pipeline, responsible for isolating site-specific behavior and dispatching extraction logic based on URL patterns. It serves as a modular dispatch layer that allows the system to handle specialized content structures for known platforms—such as GitHub, Substack, and Reddit—while falling back to generic heuristic extraction for unknown sites.

The registry is designed to keep acquisition concerns (fetching HTML) separate from extraction concerns (parsing content). It provides a stable interface for the orchestration layer to request content that has been pre-processed according to site-specific rules, ensuring higher fidelity in the resulting Markdown output.

Sources: extensions/swe_distiller/ARCHITECTURE.md:3-15, extensions/swe_distiller/src/extractors/registry.rs:24-32

Architecture and Components

The Extractor Registry operates as a collection of ExtractorEntry structures, each mapping specific URL patterns to a synchronization extraction function. The registry is initialized as a static vector of these entries using once_cell::sync::Lazy.

Registry Data Structures

The registry utilizes the following primary data structures to manage the dispatch logic:

Sources: extensions/swe_distiller/src/extractors/registry.rs:9-17, extensions/swe_distiller/src/extractors/registry.rs:37-60

Extractor Registry Flow

The following diagram illustrates how the registry identifies and executes a matching extractor during the distillation process:

Extractor Selection Logic

The selection process is deterministic. When a URL is passed to the registry, it iterates through the EXTRACTORS list. A match is determined by the UrlPattern enum:

1. Domain Match: Parses the URL and checks if the host string matches the target domain or ends with a subdomain of that domain (e.g., github.com and www.github.com).
2. Regex Match: Executes a regular expression against the full URL string (e.g., matching Substack's specific path structures).

Sources: extensions/swe_distiller/src/extractors/registry.rs:44-58

Supported Platforms

The registry currently includes specialized extractors for several major platforms:

• Substack: Matches substack.com or specific article/note path regex.
• GitHub: Matches github.com and targets containers like article.markdown-body.
• Reddit: Matches reddit.com.
• Hacker News: Matches news.ycombinator.com and targets table.fatitem.
• Social Media: Specialized handling for x.com, twitter.com, and youtube.com.
• Chat LLM Interfaces: Specific logic for chatgpt.com, claude.ai, grok.com, and gemini.google.com.

Sources: extensions/swe_distiller/src/extractors/registry.rs:66-118

Implementation of Site Extractors

Each site-specific extractor implements a common interface: fn extract(&Html, &str) -> Option<ExtractorResult>. These functions often use a "best candidate" approach, scanning a list of CSS selectors specific to the platform and selecting the one with the highest word count that meets a minimum threshold.

Example: GitHub Extractor logic

The GitHub extractor identifies content by checking for specific classes used in GitHub's Markdown rendering engine.

pub fn extract(html: &Html, url: &str) -> Option<ExtractorResult> {
    if !is_github_url(url) { return None; }

    let selectors = [
        "article.markdown-body",
        "div#readme article",
        "div.markdown-body",
        "div.TimelineItem-body",
    ];
    let content_html = best_candidate_html(html, &selectors, 30)?;
    Some(ExtractorResult {
        content_html,
        site: Some("GitHub".to_string()),
        ..ExtractorResult::default()
    })
}

Sources: extensions/swe_distiller/src/extractors/github.rs:6-21

Integration with Pipeline

The Registry is invoked early in the parse_internal pipeline. If the registry returns an ExtractorResult, the content_html from the extractor is preferred over the generic find_main_content_html heuristic. However, even when a specialized extractor is used, the resulting HTML still passes through standard normalization stages, including footnote standardization and removal of hidden elements or clutter patterns.

Sources: extensions/swe_distiller/src/lib.rs:142-160, extensions/swe_distiller/ARCHITECTURE.md:21-32

Pipeline Sequence Diagram

Conclusion

The Extractor Registry provides the necessary flexibility for swe_distiller to handle the diverse layouts of the modern web. By encapsulating site-specific CSS selectors and URL patterns within individual modules and a central registry, the system ensures that high-value targets like technical documentation (GitHub) or long-form articles (Substack) are extracted with maximum precision while maintaining a robust fallback for the general web.