Document Fetching & Browser Fallback

Relevant source files

The following files were used as context for generating this wiki page:

• extensions/swe_distiller/src/fetch.rs
• extensions/swe_distiller/ARCHITECTURE.md
• extensions/swe_distiller/src/lib.rs
• extensions/swe_distiller/README.md
• extensions/swe_distiller/src/main.rs
• extensions/swe_distiller/Cargo.toml

The Document Fetching & Browser Fallback system in swe_distiller is responsible for the initial acquisition of web content. It serves as the entry point of the extraction pipeline, managing HTTP requests, retries, character set decoding, and specialized fallbacks for anti-bot measures. The primary goal is to return a decoded HTML string to the orchestration layer for further processing.

This module is designed to be deterministic and fast by default, isolating site-specific acquisition concerns from the core extraction logic. It supports a feature-flagged browser fallback using chromiumoxide to handle complex pages or challenges that standard HTTP clients cannot resolve.

Sources: extensions/swe_distiller/ARCHITECTURE.md:7-14, extensions/swe_distiller/src/lib.rs:25-34

Acquisition Architecture

The fetching layer utilizes reqwest for standard HTTP acquisition and supports proxy configuration, custom User-Agents, and specific language headers. It follows a multi-attempt retry strategy for transient failures.

The Fetch Pipeline

The flow begins with identifying the appropriate User-Agent based on the target domain. For instance, GitHub specifically receives a "bot" UA string. The system then attempts up to three HTTP GET requests, applying an exponential backoff delay between retries for specific status codes like 429 (Too Many Requests) or 503 (Service Unavailable).

The following diagram illustrates the high-level decision logic during a fetch operation:

A flowchart showing the logical progression from initial request to either success, fallback, or failure.
Sources: extensions/swe_distiller/src/fetch.rs:26-38, extensions/swe_distiller/src/fetch.rs:40-128

Key Acquisition Parameters

Parameter	Default Value / Behavior	Source
MAX_SIZE	5 MB	fetch.rs:16
FETCH_TIMEOUT	10 Seconds	fetch.rs:17
MAX_FETCH_ATTEMPTS	3	fetch.rs:18
DEFAULT_UA	Mozilla/5.0 (compatible; swe_distiller/1.0; ...)	fetch.rs:20
BOT_UA_DOMAINS	["github.com"]	fetch.rs:23

Browser Fallback Mechanism

When a standard HTTP request fails with a 403 Forbidden status, the system triggers fallback mechanisms. The most robust fallback is the Browser Fallback, which is only available if the project is built with the browser feature flag.

Browser Execution Flow

The browser fallback uses a headless Chromium instance (via chromiumoxide) to load the page, allowing JavaScript to execute and potentially bypass basic anti-bot challenges. If the resulting page title or HTML contains specific "challenge" markers (e.g., "Just a moment", "cf-browser-verification", or "captcha"), the fallback is considered failed to prevent returning garbage data.

A sequence diagram detailing the interaction between the fetch layer and the headless browser during a fallback operation.
Sources: extensions/swe_distiller/src/fetch.rs:141-167, extensions/swe_distiller/src/fetch.rs:132-139, extensions/swe_distiller/Cargo.toml:11-12

Feature Flagging

The browser dependency is optional to keep the core binary light.

• Build Command: cargo build --features browser
• Run Command: cargo run --features browser -- <url>

Sources: extensions/swe_distiller/README.md:43-47, extensions/swe_distiller/Cargo.toml:12

Content Validation and Decoding

After a successful acquisition, the system performs strict validation before passing data to the extractor.

Size and Type Constraints

• Content-Type Check: The system verifies that the response is either text/html or application/xhtml+xml. If not, it returns an error.
• Size Limits: The system checks both the content-length header and the actual downloaded byte length against the MAX_SIZE limit (5MB) to prevent memory exhaustion.

Sources: extensions/swe_distiller/src/fetch.rs:88-104

Character Set Detection

The detect_charset function ensures the bytes are correctly decoded into a Rust String. It follows this priority:

1. HTTP Header: Looks for charset= in the Content-Type header.
2. HTML Meta Tag: Scans the first 2048 bytes of the document for a <meta charset="..."> declaration.
3. Fallback: Defaults to UTF-8.

Sources: extensions/swe_distiller/src/fetch.rs:175-197

Site-Specific Fallbacks: Medium RSS

For medium.com and its subdomains, the fetcher includes a specialized fallback that attempts to retrieve content via RSS feeds when a direct HTTP request is blocked (403).

Medium Logic

1. URL Normalization: Strips query parameters and fragments.
2. ID Extraction: Extracts the Medium post ID (e.g., bdaf1bd6e64b).
3. Feed Discovery: Generates candidate feed URLs (e.g., https://medium.com/feed/@user).
4. Content Wrapping: If the target post is found in the RSS XML, the content:encoded or description field is wrapped in a basic HTML structure to be processed by the standard extraction pipeline.

Sources: extensions/swe_distiller/src/fetch.rs:219-270, extensions/swe_distiller/src/fetch.rs:364-375

Conclusion

The Document Fetching & Browser Fallback system provides a resilient foundation for swe_distiller. By combining standard HTTP retries with specialized site fallbacks and a feature-flagged headless browser, it ensures high acquisition success rates across diverse web environments while maintaining strict limits on resource consumption and data validity.