Developer Platform Extractors
Relevant source files
The following files were used as context for generating this wiki page:
Developer Platform Extractors are specialized modules within the swe_distiller pipeline designed to handle site-specific HTML structures from platforms like GitHub and Substack. These extractors isolate site-specific behavior to ensure that the primary content acquisition path remains deterministic and fast, while still allowing for high-fidelity extraction of complex developer-centric pages.
The extractors function as a modular layer between the fetch phase and the general content cleanup stages. They provide a contract where specific URL patterns trigger targeted extraction logic, returning purified HTML and optional metadata overrides to the orchestration pipeline.
Sources: extensions/swe_distiller/ARCHITECTURE.md:3-15, extensions/swe_distiller/src/lib.rs:32-45
Architecture and Integration
The extractor system is integrated into the swe_distiller library's orchestration logic. During the pipeline execution, the system attempts to match the current URL against a registry of site-specific extractors before falling back to heuristic content discovery.
Extraction Flow
The following diagram illustrates how extractors fit into the high-level content processing flow:
The pipeline prioritizes site-specific extractors to ensure that the most relevant container (e.g., a README on GitHub) is selected over generic body tags.
Sources: extensions/swe_distiller/ARCHITECTURE.md:21-31, extensions/swe_distiller/src/lib.rs:107-130
GitHub Extractor Implementation
The GitHub extractor focuses on identifying and isolating documentation and technical content within the GitHub ecosystem. It primarily targets markdown-body elements, which are standard for READMEs and wiki pages.
Logic and Selection
The extractor validates the URL to ensure it belongs to the github.com domain or a .github.com subdomain. Once validated, it iterates through a prioritized list of CSS selectors to find the "best" candidate based on word count.
| Selector | Target Content |
|---|---|
article.markdown-body | Standard README and Wiki content containers |
div#readme article | Readme-specific wrappers |
div.markdown-body | Generic markdown containers |
div.TimelineItem-body | Issue/Pull Request discussion bodies |
Sources: extensions/swe_distiller/src/extractors/github.rs:7-22
Candidate Evaluation
The extractor uses a helper function, best_candidate_html, which enforces a minimum threshold of 30 words to prevent capturing small navigational snippets or UI fragments. If multiple nodes match a selector, the one with the highest word count is selected.
Sources: extensions/swe_distiller/src/extractors/github.rs:35-51
Asynchronous and Registry Dispatch
The library provides two primary ways to run extractors: synchronous and asynchronous. Asynchronous extractors are preferred when dealing with platforms that might require supplementary lookups or complex parsing.
Data Structures
The extractors communicate with the pipeline using the ExtractorResult structure, which contains the cleaned HTML and various metadata fields that can override generic metadata extraction.
| Field | Type | Description |
|---|---|---|
content_html | String | The purified HTML fragment extracted from the page. |
title | Option<String> | Site-specific title (e.g., from Substack preloads). |
author | Option<String> | Overridden author information. |
site | Option<String> | The name of the platform (e.g., "GitHub", "Substack"). |
published | Option<String> | Specific publication timestamp from platform metadata. |
Sources: extensions/swe_distiller/src/lib.rs:188-212, extensions/swe_distiller/tests/integration.rs:175-185
Platform Specifics
Substack Preload Extractor
As demonstrated in integration tests, certain platforms like Substack are handled via script preload extractors. These do not just parse the DOM but instead extract content from JSON objects stored in window._preloads or similar script tags. This allows the extractor to obtain the raw body_html and precise metadata that might be difficult to parse from the rendered DOM.
Sources: extensions/swe_distiller/tests/integration.rs:163-195
X (Twitter) Extractor
The pipeline includes support for social developer platforms like X. The extractor targets specific testing IDs such as tweetText to isolate the core content of a thread or post, filtering out the surrounding social graph and navigation chrome.
Sources: extensions/swe_distiller/tests/integration.rs:211-223
Summary
Developer Platform Extractors provide the necessary specialization for swe_distiller to achieve high accuracy on complex sites. By isolating these rules in the extractors/ directory, the project maintains a clean separation between generic heuristic extraction and platform-specific logic. This modularity allows for rapid expansion to new platforms (like YouTube or Hacker News) without increasing the complexity of the core extraction pipeline.
Sources: extensions/swe_distiller/ARCHITECTURE.md:73-80, extensions/swe_distiller/TODO.md:27-35