Getting Started & Setup
Relevant source files
The following files were used as context for generating this wiki page:
swe_distiller is a high-performance Rust-based extraction pipeline and CLI utility designed to transform web URLs into clean, structured Markdown or JSON content. It serves as an extension within the broader swe-term harness, providing deterministic heuristic extraction with optional Large Language Model (LLM) augmentation.
The system is architected to keep the default execution path fast and deterministic while isolating site-specific behaviors into modular extractors. It supports advanced features such as browser-based fallback for complex pages and multi-provider LLM pipelines for high-quality content distillation.
Sources: extensions/swe_distiller/README.md:3-5, extensions/swe_distiller/ARCHITECTURE.md:3-10, extensions/swe_distiller/src/lib.rs:26-36
Environment Prerequisites & Installation
To set up swe_distiller, users must have the Rust toolchain installed. The project utilizes cargo for building and dependency management.
Build Commands
# Build the project
cargo build
# Run the project with a specific URL
cargo run -- "https://example.com/article" -o webpage.md
Sources: extensions/swe_distiller/README.md:10-18
Optional Features
The tool includes a browser-based fallback feature for handling JavaScript-heavy websites. This is enabled via a specific feature flag during compilation:
cargo run --features browser -- "https://example.com/article" -o webpage.md
Sources: extensions/swe_distiller/README.md:32-34, extensions/swe_distiller/Cargo.toml:8-9
Configuration & Runtime Modes
swe_distiller supports multiple runtime modes that determine how content is extracted and formatted. These are controlled via CLI arguments or environment variables.
CLI Configuration Options
| Flag | Description | Default |
|---|---|---|
--mode | Output format: markdown or json. | markdown |
--llm | Enables the LLM extraction pipeline. | false |
--stdout | Writes output to stdout instead of a file. | false |
--lang | Preferred language (BCP 47). | None |
--proxy | Optional outbound HTTP/HTTPS proxy URL. | None |
--debug | Enables structured debug logging to stderr. | false |
Sources: extensions/swe_distiller/src/main.rs:17-50, extensions/swe_distiller/src/lib.rs:27-30
LLM Pipeline Environment Variables
When the --llm mode is enabled, the system looks for specific environment variables to configure providers and timeouts.
| Variable | Description | Default |
|---|---|---|
SWE_DISTILLER_LLM_PROVIDERS | Comma-separated list of LLM providers. | gemma4:31b-cloud,reader-lm:1.5b,jina |
SWE_DISTILLER_LLM_TIMEOUT_MS | Timeout for LLM provider requests. | 60000 |
SWE_DISTILLER_LLM_MAX_INPUT_CHARS | Maximum characters sent to the LLM. | 60000 |
OLLAMA_BASE_URL | Base URL for Ollama local provider. | http://127.0.0.1:11434 |
JINA_API_KEY | API key for the Jina.ai reader service. | None |
Sources: extensions/swe_distiller/README.md:24-28, extensions/swe_distiller/src/extraction/llm.rs:11-13, extensions/swe_distiller/src/extraction/llm.rs:71-92
System Architecture & Data Flow
The swe_distiller pipeline follows a structured sequence from URL acquisition to final markdown conversion.
Pipeline Execution Flow
The diagram below illustrates the high-level logic used when a URL is processed.
The pipeline begins with URL acquisition, potentially using a browser fallback. It then attempts site-specific extraction before falling back to general heuristics. If LLM mode is active, the LLM output is compared against the heuristic baseline using safety and noise checks before acceptance.
Sources: extensions/swe_distiller/ARCHITECTURE.md:15-24, extensions/swe_distiller/src/lib.rs:26-66, extensions/swe_distiller/src/extraction/llm.rs:25-58
LLM Provider Orchestration
When multiple LLM providers are configured, they are tried in a specific sequence until a candidate passes all validation gates.
The extract_via_llm function handles this orchestration, applying passes_safety_checks and structure_noise_score to ensure the generated content is high quality and represents the original text accurately.
Sources: extensions/swe_distiller/src/extraction/llm.rs:25-58, extensions/swe_distiller/src/extraction/checks.rs:41-65
Core Components & Modules
The source code is organized into discrete modules to separate acquisition, extraction, and formatting concerns.
| Module | File Path | Responsibility |
|---|---|---|
| CLI | src/main.rs | Handles command-line arguments and file I/O. |
| Orchestration | src/lib.rs | Manages the overall pipeline and recovery plans. |
| Metadata | src/metadata.rs | Extracts title, author, date, and site info from HTML and Schema.org. |
| LLM Orchestration | src/extraction/llm.rs | Communicates with LLM providers (Jina, Ollama). |
| Safety Checks | src/extraction/checks.rs | Scores content quality and validates LLM output. |
| Markdown | src/markdown_ast.rs | Converts DOM structures to Markdown using an AST approach. |
| Standardization | src/standardize/ | Normalizes content structure (footnotes, callouts). |
Sources: extensions/swe_distiller/ARCHITECTURE.md:27-41, extensions/swe_distiller/src/lib.rs:1-16
Summary of Setup
Setting up swe_distiller involves building the Rust binary and optionally configuring LLM provider credentials. The tool defaults to a fast, heuristic-based extraction but can be extended with browser support for complex pages or LLM support for higher quality distillation. Key configuration is handled via environment variables like SWE_DISTILLER_LLM_PROVIDERS and CLI flags like --llm and --debug.
Sources: extensions/swe_distiller/README.md:1-35, extensions/swe_distiller/ARCHITECTURE.md:68-81