Social Media Extractors
Relevant source files
The following files were used as context for generating this wiki page:
Social Media Extractors are specialized components within the swe_distiller pipeline designed to isolate site-specific content and metadata from platforms like X (formerly Twitter), Reddit, GitHub, and Hacker News. By isolating platform-specific behavior, the system ensures that high-signal content—such as tweet text, forum threads, or README files—is accurately captured while ignoring the surrounding navigational "chrome" and UI clutter common to social platforms.
These extractors are integrated into the main pipeline via a centralized registry, which routes incoming HTML documents to the appropriate extraction logic based on URL patterns. When a site-specific match is found, the extractor provides an ExtractorResult containing the cleaned HTML and potential metadata overrides, which the pipeline prioritizes over generic heuristic discovery.
Sources: extensions/swe_distiller/ARCHITECTURE.md:5-15, extensions/swe_distiller/src/extractors/registry.rs:43-90
Architecture and Registry Flow
The extractor system operates as a pluggable layer between document acquisition and content normalization. The registry.rs module manages a collection of ExtractorEntry structures, each mapping URL patterns (domain strings or regex) to specific synchronous extraction functions.
The registry determines the appropriate extractor by checking the domain or matching against pre-defined regular expressions. For instance, Substack uses a specific path regex to identify notes and posts, while X and Reddit rely on domain-level matching.
Sources: extensions/swe_distiller/src/extractors/registry.rs:18-41, extensions/swe_distiller/ARCHITECTURE.md:21-30
Extractor Dispatch Table
| Platform | Pattern Type | Implementation Module | Target Content |
|---|---|---|---|
| Substack | Domain / Regex | substack::extract | Preloaded JSON/HTML body |
| Domain | reddit::extract | Thread content & comments | |
| GitHub | Domain | github::extract | READMEs & Timeline items |
| Hacker News | Domain | hackernews::extract | Comment trees |
| X / Twitter | Domain | x_twitter::extract | Tweet text (tweetText) |
| YouTube | Domain | youtube::extract | Transcripts / Descriptions |
| Chat LLMs | Domain | chat_llm::extract | Conversation turns |
Sources: extensions/swe_distiller/src/extractors/registry.rs:64-106, extensions/swe_distiller/tests/integration.rs:218-230
Implementation Details
GitHub Extractor logic
The GitHub extractor specifically targets documentation and technical content within the repository interface. It utilizes a prioritized list of CSS selectors to find the "best candidate" for content, ensuring that large README files or issue timeline items are captured rather than sidebar navigation.
let selectors = [
"article.markdown-body",
"div#readme article",
"div.markdown-body",
"div.TimelineItem-body",
];
Sources: extensions/swe_distiller/src/extractors/github.rs:10-15
The best_candidate_html function iterates through these selectors and selects the node with the highest word count that meets a minimum threshold (default 30 words), ensuring signal over noise.
Sources: extensions/swe_distiller/src/extractors/github.rs:29-47
Data Structures
The primary interface between an extractor and the main pipeline is the ExtractorResult struct. This structure allows extractors to not only return the core content but also specify metadata that might be difficult to parse via standard meta tags on specific platforms.
pub struct ExtractorResult {
pub content_html: String,
pub title: Option<String>,
pub description: Option<String>,
pub author: Option<String>,
pub published: Option<String>,
pub site: Option<String>,
}
Sources: extensions/swe_distiller/src/extractors/registry.rs:8-16
Integration with Pipeline
The lib.rs module orchestrates how these extractors are invoked. The pipeline first attempts to run "async preferred" extractors (useful for API-backed sources like YouTube transcripts). If no extractor provides content, the system falls back to a heuristic find_main_content_html method.
Metadata Priority
When an extractor provides metadata (title, author, etc.), the pipeline uses a choose_override utility to prioritize the extractor's findings over generic metadata.rs extraction logic. This is critical for sites like X or Reddit where the "author" might be a specific handle found in a data attribute rather than a standard <meta name="author"> tag.
Sources: extensions/swe_distiller/src/lib.rs:252-275, extensions/swe_distiller/src/metadata.rs:72-88
Conclusion
Social Media Extractors provide the necessary specialization to handle the highly structured yet cluttered nature of modern social platforms. By utilizing a registry-based dispatch system and targeted CSS selection, swe_distiller maintains high extraction quality across diverse domains like GitHub READMEs, Reddit threads, and X posts.