swe_distiller Architecture
Relevant source files
The following files were used as context for generating this wiki page:
swe_distiller is a Rust-first extraction pipeline and CLI utility designed to transform a URL or raw HTML into clean, readable article content, primarily in Markdown format. It serves as an independent extraction component within the swe-term project, prioritizing speed, determinism, and modularity in its design.
Sources: extensions/swe_distiller/ARCHITECTURE.md:3-5, extensions/swe_distiller/README.md:3-5
The system follows a layered approach that separates URL acquisition, content extraction, and output conversion. While the default path relies on deterministic heuristics for performance, the architecture supports advanced modes such as browser fallback for dynamic pages and an optional LLM-driven extraction pipeline for high-fidelity overrides.
Sources: extensions/swe_distiller/ARCHITECTURE.md:7-27, extensions/swe_distiller/src/lib.rs:25-50
System Overview and High-Level Flow
The swe_distiller pipeline orchestrates several stages to convert a raw URL into structured data. It begins with a fetch layer that handles HTTP acquisition and charset decoding, followed by a parsing stage where site-specific extractors or general heuristic logic are selected. The resulting HTML is then cleaned via multiple removal passes before being converted into Markdown through a DOM-walking AST converter.
Sources: extensions/swe_distiller/ARCHITECTURE.md:16-27, extensions/swe_distiller/src/lib.rs:188-250
Processing Pipeline Flow
The following diagram illustrates the sequence of operations within the distillation pipeline:
The pipeline includes built-in recovery plans that automatically relax extraction constraints (e.g., re-enabling hidden elements or partial selectors) if initial word counts are too low.
Sources: extensions/swe_distiller/ARCHITECTURE.md:20-27, extensions/swe_distiller/src/lib.rs:88-106, extensions/swe_distiller/src/lib.rs:328-356
Core Components
Pipeline Orchestration (lib.rs)
The central coordinator is the ParsePipeline struct, which manages the lifecycle of a single extraction request. It handles the initial extraction pass and executes recovery plans if the result does not meet quality thresholds.
Sources: extensions/swe_distiller/src/lib.rs:81-125
Extraction Engine
The system uses three primary methods for locating content:
- Site-Specific Extractors: Dispatch based on URL patterns for high-accuracy extraction (e.g., Substack, GitHub).
- Heuristic Discovery: A fallback that analyzes DOM structure to identify the "main" content block.
- LLM Extraction: An optional asynchronous path that uses external providers (like Ollama or Jina) to re-generate the article content.
Sources: extensions/swe_distiller/ARCHITECTURE.md:46-51, extensions/swe_distiller/src/lib.rs:194-200, extensions/swe_distiller/src/extraction/llm.rs:22-54
Data Structures
The following table summarizes the primary data models used within the pipeline:
| Structure | Purpose | Source |
|---|---|---|
DistillerOptions | Configuration for extraction (LLM flags, debug, proxy, etc.) | src/types.rs |
DistillerResponse | Final output containing content, markdown, metadata, and timing. | src/types.rs |
ExtractorResult | Intermediate result from a site-specific extractor. | src/lib.rs:18-20 |
RecoveryPlan | Defines conditions for re-running extraction with relaxed filters. | src/lib.rs:73-78 |
Sources: extensions/swe_distiller/src/lib.rs:23-24, extensions/swe_distiller/src/lib.rs:73-78, extensions/swe_distiller/ARCHITECTURE.md:53-56
LLM Extraction Pipeline
When the --llm flag is enabled, swe_distiller attempts to use configured LLM providers to improve extraction quality. This process is isolated from the main fetch logic and includes strict safety checks to prevent hallucinations or excessive content loss.
Sources: extensions/swe_distiller/src/extraction/llm.rs:22-54, extensions/swe_distiller/src/lib.rs:40-62
LLM Interaction Sequence
LLM providers are attempted in the order specified by the SWE_DISTILLER_LLM_PROVIDERS environment variable (defaulting to gemma4:31b-cloud,reader-lm:1.5b,jina).
Sources: extensions/swe_distiller/src/extraction/llm.rs:15, extensions/swe_distiller/src/extraction/llm.rs:65-71
Metadata and Content Standardization
Metadata Extraction (metadata.rs)
The system extracts rich metadata from multiple sources, including JSON-LD (Schema.org) blocks and standard OpenGraph/Twitter meta tags.
- Title/Author/Date: Prioritizes Schema.org data, then meta tags, and finally heuristics from the Markdown itself.
- Domain/Favicon: Derived from the source URL and site meta tags.
Sources: extensions/swe_distiller/src/metadata.rs:37-133
Markdown Conversion (markdown_ast.rs)
The html_to_markdown function utilizes a DOM-walking strategy to convert cleaned HTML into Markdown.
- AST Mapping: Maps HTML tags to Markdown equivalents (e.g.,
<h1>to#,<ul>to lists). - Code Block Detection: Attempts to infer programming languages from CSS classes (e.g.,
language-rust). - Table Support: Converts HTML
<table>structures into Markdown pipes.
Sources: extensions/swe_distiller/src/markdown_ast.rs:17-128, extensions/swe_distiller/src/markdown_ast.rs:242-261
Conclusion
The swe_distiller architecture provides a robust and extensible framework for web content extraction. By combining fast heuristic methods with an optional LLM-driven pipeline and a modular extractor registry, it balances performance and accuracy. The use of structured recovery plans and safety gates ensures that the system provides high-quality Markdown output even for complex or poorly structured web pages.
Sources: extensions/swe_distiller/ARCHITECTURE.md:7-14, extensions/swe_distiller/src/lib.rs:328-356