swe_distiller Pipeline Overview
Relevant source files
The following files were used as context for generating this wiki page:
swe_distiller is a Rust-first content extraction pipeline and CLI designed to transform raw web URLs into clean, readable Markdown or JSON. It serves as a specialized tool within the swe-term harness to provide high-quality input for AI agents by stripping away navigation, advertisements, and other "chrome" while preserving core article content and metadata.
The pipeline is architected to be deterministic and fast by default, utilizing heuristic extraction, while offering an optional LLM-augmented mode for complex layouts. It follows a modular design that separates acquisition (fetching) from extraction logic and output conversion.
Sources: extensions/swe_distiller/ARCHITECTURE.md:3-10, extensions/swe_distiller/README.md:3-5
High-Level Architecture
The extraction process follows a linear progression from URL acquisition to final document generation. The system is divided into functional modules that handle specific stages of the transformation.
Execution Flow
The following diagram illustrates the standard data flow through the swe_distiller modules:
The flow begins with HTTP acquisition and charset decoding, moves through content selection and cleanup, and concludes with structural normalization and metadata enrichment.
Sources: extensions/swe_distiller/ARCHITECTURE.md:15-25, extensions/swe_distiller/src/lib.rs:30-76
Pipeline Core Stages
1. Acquisition and Parsing
The pipeline starts by fetching raw HTML using a specialized fetch layer that handles HTTP retries and charset decoding. When the --features browser flag is enabled, it can fall back to a headless browser for JavaScript-heavy pages.
Sources: extensions/swe_distiller/ARCHITECTURE.md:33-37, extensions/swe_distiller/src/lib.rs:32-40
2. Extraction & Removal
The extraction stage identifies the main content block and applies several pruning passes to remove clutter:
- Site-Specific Extractors: Targeted logic for sites like Substack, GitHub, or X.
- Removal Passes: Systematic deletion of hidden elements, small images, low-scoring blocks, and known clutter patterns (e.g., newsletter signups, sharing buttons).
- Standardization: Normalizing footnotes and callouts within the HTML structure.
Sources: extensions/swe_distiller/src/lib.rs:163-228, extensions/swe_distiller/src/removal/content_patterns.rs:36-50
3. Metadata Extraction
Metadata is gathered from multiple sources including OpenGraph tags, JSON-LD (Schema.org), and heuristic analysis of the content itself (e.g., finding bylines in the first 20 lines of text).
Sources: extensions/swe_distiller/src/metadata.rs:43-160, extensions/swe_distiller/src/lib.rs:238-270
4. LLM Extraction Pipeline
When the --llm flag is used, the pipeline attempts to refine the output using configured providers (e.g., Ollama, Jina). This stage includes safety checks to ensure the LLM output is not hallucinated or overly truncated.
The LLM pipeline uses structure_noise_score and passes_safety_checks to validate candidates against a heuristic baseline.
Sources: extensions/swe_distiller/src/extraction/llm.rs:27-66, extensions/swe_distiller/src/extraction/checks.rs:41-61
Key Configuration and Types
The pipeline behavior is controlled via the DistillerOptions struct, while results are returned in a DistillerResponse.
Distiller Options
| Option | Type | Description |
|---|---|---|
mode | ParseMode | Determines if output is Markdown or Json. |
llm | bool | Enables the LLM-augmented extraction path. |
debug | bool | Emits structured JSON logs to stderr for observability. |
language | Option<String> | Preferred BCP 47 language code for fetching. |
remove_hidden_elements | bool | Whether to prune elements with display: none or hidden attributes. |
Sources: extensions/swe_distiller/src/main.rs:56-65, extensions/swe_distiller/src/lib.rs:188-196
LLM Provider Environment Variables
The LLM pipeline is configured via environment variables to allow flexible provider switching.
| Variable | Default Value | Description |
|---|---|---|
SWE_DISTILLER_LLM_PROVIDERS | gemma4:31b-cloud,reader-lm:1.5b,jina | Comma-separated list of LLM models to attempt. |
SWE_DISTILLER_LLM_TIMEOUT_MS | 60000 | Max time allowed for an LLM response. |
OLLAMA_BASE_URL | http://127.0.0.1:11434 | Endpoint for local Ollama providers. |
Sources: extensions/swe_distiller/src/extraction/llm.rs:13-16, extensions/swe_distiller/src/extraction/llm.rs:77-100
Observability and Recovery
The pipeline includes a "Recovery" mechanism. If the initial extraction pass results in a word count below a specific threshold (e.g., 200 words), it automatically retries with relaxed removal settings.
Sources: extensions/swe_distiller/src/lib.rs:125-142, extensions/swe_distiller/src/lib.rs:309-335
// Example of a Recovery Plan defined in lib.rs
fn recovery_plans(opts: &DistillerOptions) -> Vec<RecoveryPlan> {
let mut plans = Vec::with_capacity(3);
let mut partial_selector_retry = opts.clone();
partial_selector_retry.remove_partial_selectors = false;
plans.push(RecoveryPlan {
minimum_word_count: 200,
options: partial_selector_retry,
require_double_improvement: true,
});
// ... further plans
plans
}
Sources: extensions/swe_distiller/src/lib.rs:309-325
Conclusion
The swe_distiller pipeline provides a robust framework for converting noisy HTML into clean, structured Markdown. By combining deterministic heuristic passes with optional, safety-gated LLM extraction, it ensures high-fidelity content acquisition for downstream developer tools and AI agents.