LLM Content Extraction Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• extensions/swe_distiller/src/extraction/llm.rs
• extensions/swe_distiller/src/extraction/checks.rs
• extensions/swe_distiller/src/lib.rs
• extensions/swe_distiller/ARCHITECTURE.md
• extensions/swe_distiller/src/main.rs
• extensions/swe_distiller/src/markdown.rs

The LLM Content Extraction Pipeline is an optional, non-deterministic path within swe_distiller designed to enhance article extraction by leveraging Large Language Models. While the default pipeline uses fast, deterministic heuristics to identify and clean content, the LLM mode provides an override mechanism that uses configured AI providers to convert raw HTML into clean, structured Markdown.

This pipeline operates with a "safety-first" design, applying rigorous quality gates and heuristic fallbacks. If an LLM candidate fails safety checks or exceeds noise thresholds compared to the heuristic baseline, the system automatically reverts to the output of the standard heuristic pipeline to ensure reliability.

Sources: extensions/swe_distiller/ARCHITECTURE.md:46-52, extensions/swe_distiller/src/lib.rs:43-69

Pipeline Architecture and Data Flow

The LLM pipeline is triggered when the --llm flag is provided to the CLI. It runs in parallel or as a staged override to the standard heuristic extraction. The process involves preprocessing raw HTML to fit context windows, querying a list of providers (such as Jina or local Ollama instances), and validating the resulting Markdown against the original heuristic output.

High-Level Execution Flow

The following diagram illustrates how the LLM extraction integrates into the broader swe_distiller pipeline.

The pipeline attempts providers in order and validates them against a heuristic baseline.
Sources: extensions/swe_distiller/src/lib.rs:30-75, extensions/swe_distiller/src/extraction/llm.rs:25-63

Provider Orchestration

The pipeline supports multiple providers configured via environment variables. It follows a sequential trial-and-error approach where the first provider to produce an acceptable result terminates the search.

Supported Provider Types

• Jina Reader: An external API (r.jina.ai) that performs remote extraction.
• Ollama: Local LLM instances used for precise HTML-to-Markdown conversion using models like reader-lm or gemma.

Sources: extensions/swe_distiller/src/extraction/llm.rs:114-171

Configuration Parameters

Variable	Type	Default	Description
SWE_DISTILLER_LLM_PROVIDERS	String	gemma4:31b-cloud,reader-lm:1.5b,jina	Comma-separated list of providers to attempt.
SWE_DISTILLER_LLM_TIMEOUT_MS	u64	60000	Timeout for each provider request in milliseconds.
SWE_DISTILLER_LLM_MAX_INPUT_CHARS	usize	60000	Maximum characters of HTML sent to the LLM after preprocessing.
OLLAMA_BASE_URL	String	http://127.0.0.1:11434	Base URL for local Ollama API.
JINA_API_KEY	String	(Optional)	Bearer token for Jina API authentication.

Sources: extensions/swe_distiller/src/extraction/llm.rs:13-16, extensions/swe_distiller/src/extraction/llm.rs:77-106

Preprocessing and Cleanup

Before HTML is sent to an LLM, it undergoes heavy preprocessing to remove non-essential elements. This reduces token consumption and focuses the model on relevant content.

1. Block Removal: Tags such as <script>, <style>, <noscript>, <nav>, <footer>, and <aside> are stripped.
2. Inline Removal: Tags like <header> and <form> are removed.
3. Whitespace Normalization: Multiple consecutive newlines are collapsed to \n\n.
4. Trimming: The final string is truncated to the configured max_input_chars.

Sources: extensions/swe_distiller/src/extraction/llm.rs:175-189

Quality and Safety Gates

A critical component of the pipeline is the checks module, which ensures LLM hallucinations or failures do not degrade the output quality.

Safety Check Sequence

The safety module acts as a gatekeeper against content loss or excessive halluncination.
Sources: extensions/swe_distiller/src/extraction/checks.rs:43-70, extensions/swe_distiller/src/extraction/llm.rs:52-60

Noise Scoring Logic

The structure_noise_score function calculates a penalty score for non-article content. LLM outputs are rejected if their noise score significantly exceeds the heuristic baseline. Penalties include:

• UI Chrome (+3): Presence of terms like "Sign up", "Search", or "Share".
• Fragmentary Text (+1): Lines with 2 or fewer words that are not headings.
• Excessive Links (+1): Short lines formatted entirely as links.
• Structural Clutter (+2): Lines containing only symbols like ! or ·.

Sources: extensions/swe_distiller/src/extraction/checks.rs:1-38

Integration and Implementation

The pipeline integration occurs in extract_url within lib.rs. If the LLM produces a valid candidate, apply_llm_markdown_override is used to update the DistillerResponse.

// extensions/swe_distiller/src/lib.rs:343-359
fn apply_llm_markdown_override(result: &mut DistillerResponse, title_hint: &str, markdown: &str) {
    let markdown = markdown::ensure_title_heading(markdown, title_hint);
    let markdown = markdown.trim().to_string();
    if markdown.is_empty() {
        return;
    }

    if let Some(title) = markdown::guess_title(&markdown).filter(|title| !title.trim().is_empty()) {
        result.title = title;
    }

    result.word_count = dom::count_words(&markdown);
    result.description = markdown_description(&markdown);
    result.content = markdown.clone();
    result.content_markdown = Some(markdown);
}

Sources: extensions/swe_distiller/src/lib.rs:343-359

Summary

The LLM Content Extraction Pipeline provides a sophisticated, multi-provider approach to article cleaning. By utilizing preprocessing to manage context limits and strict heuristic-based safety checks, it balances the power of LLMs with the reliability of deterministic parsing, ensuring high-quality Markdown output for downstream SWE-agent consumption.