Media Content Extractors
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Media Content Extractors within the swe_distiller project are site-specific modules designed to isolate specialized parsing logic for diverse web platforms. Their primary purpose is to transform platform-specific HTML or data structures (such as Substack preloads or YouTube metadata) into clean, standardized Markdown content. This modular approach allows the core pipeline to remain deterministic and fast while delegating complex, site-specific extraction concerns to specialized handlers.
Sources: extensions/swe_distiller/ARCHITECTURE.md:1-12, extensions/swe_distiller/src/lib.rs:136-140
Architecture and Pipeline Integration
The extraction system operates as a pluggable layer within the broader swe_distiller pipeline. When a URL is processed, the system first attempts to identify and run a matching site-specific extractor from a registry before falling back to generic heuristic extraction.
Extraction Data Flow
The flow below illustrates how a URL is routed through acquisition, extractor selection, and eventual content conversion.
The pipeline prefers asynchronous extractors when available, dispatching based on URL patterns.
Sources: extensions/swe_distiller/ARCHITECTURE.md:15-25, extensions/swe_distiller/src/lib.rs:37-45
The Extractor Contract
Each extractor follows a specific contract, returning an ExtractorResult which includes the extracted HTML content and optional metadata overrides. This allows extractors to provide high-fidelity data that heuristic methods might miss, such as specific authors, publication dates, or site names.
| Field | Description |
|---|---|
content_html | The raw HTML fragment containing the primary media or article content. |
title | Optional override for the document title. |
author | Optional override for the content creator. |
published | Optional override for the publication timestamp. |
site | Optional override for the platform name (e.g., "Substack"). |
Sources: extensions/swe_distiller/ARCHITECTURE.md:46-51, extensions/swe_distiller/src/lib.rs:206-240
Specific Media Extractors
Substack Extractor
Substack extraction utilizes a specialized "preload" mechanism. Rather than parsing the rendered DOM, it identifies and parses a JSON payload found within window._preloads in the HTML source. This payload contains structured feedData including the post title, subtitle, and full body_html.
Sources: extensions/swe_distiller/tests/integration.rs:188-212
Social and Development Platforms
The project includes specialized logic for various high-traffic platforms to ensure structural integrity:
- GitHub: Targets the
.markdown-bodyclass to extract clean documentation content. - X (Twitter): Focuses on
data-testid="tweetText"to capture thread content while bypassing platform chrome. - Medium: Implements a robust RSS feed fallback. If a standard HTTP fetch fails (e.g., via 403 Forbidden), the system attempts to fetch the content through Medium's RSS feeds by normalizing the URL and matching Post IDs.
Sources: extensions/swe_distiller/src/fetch.rs:106-125, extensions/swe_distiller/tests/integration.rs:228-251
Acquisition and Fallbacks
Media extraction is supported by a sophisticated fetch layer that handles user-agent spoofing and browser fallbacks.
Browser Fallback Mechanism
For platforms that implement heavy anti-bot measures, the extractor pipeline can utilize a browser-based acquisition method (if enabled via the browser feature).
Metadata Resolution Priority
The system employs a strict hierarchy for resolving metadata like author and published date, prioritizing specialized extractor results over generic tags.
- Extractor Overrides: Values explicitly returned by the site-specific extractor.
- Schema.org: Data found in
application/ld+jsonblocks. - Meta Tags: Standard tags like
og:titleortwitter:author. - Heuristics: Content-based hints (e.g., "By ..." lines in the first 20 lines of Markdown).
Sources: extensions/swe_distiller/src/metadata.rs:48-110, extensions/swe_distiller/src/lib.rs:206-235
Conclusion
Media Content Extractors provide the necessary specialization for swe_distiller to handle the complexities of modern web platforms. By isolating site-specific logic—from Substack JSON preloads to Medium RSS fallbacks—and integrating them into a unified pipeline with robust acquisition and metadata resolution, the system ensures high-quality Markdown output regardless of the source platform's underlying structure.