Environment Variables & Limits

Relevant source files

The following files were used as context for generating this wiki page:

• extensions/swe_distiller/README.md
• extensions/swe_distiller/src/fetch.rs
• extensions/swe_distiller/src/extraction/llm.rs
• extensions/swe_distiller/src/lib.rs
• extensions/swe_distiller/ARCHITECTURE.md
• extensions/swe_distiller/src/extraction/checks.rs

Introduction

The swe_distiller extension utilizes a combination of environment variables and hardcoded operational limits to manage content acquisition, Large Language Model (LLM) orchestration, and extraction quality. These configurations ensure system stability by preventing excessive resource consumption during web scraping and providing fallback mechanisms when primary extraction paths fail.

The system is designed to be deterministic by default, with elective overrides for LLM-based processing. Limits are applied at various stages, including network fetch size, request timeouts, and content similarity thresholds to maintain the integrity of distilled Markdown output.

Sources: extensions/swe_distiller/ARCHITECTURE.md, extensions/swe_distiller/README.md

LLM Configuration & Environment Variables

The LLM extraction pipeline is controlled primarily through environment variables. These variables allow users to specify provider order, authentication keys, and operational constraints for external services like Jina and local services like Ollama.

LLM Provider Settings

Variable	Description	Default Value
SWE_DISTILLER_LLM_PROVIDERS	Comma-separated list of LLM providers to attempt in order.	gemma4:31b-cloud,reader-lm:1.5b,jina
SWE_DISTILLER_LLM_TIMEOUT_MS	Global timeout for LLM provider requests.	60000 (60 seconds)
SWE_DISTILLER_LLM_MAX_INPUT_CHARS	Maximum characters of preprocessed HTML sent to the LLM.	60000
OLLAMA_BASE_URL	Base URL for local Ollama API instances.	http://127.0.0.1:11434
JINA_API_KEY	Authentication bearer token for Jina Reader API.	None

Sources: extensions/swe_distiller/src/extraction/llm.rs:13-17, extensions/swe_distiller/README.md:32-37

LLM Orchestration Flow

The following diagram illustrates how environment variables influence the selection and execution of LLM providers during the extraction process.

The orchestration logic prioritizes providers as listed in the environment. If a provider fails or exceeds the timeout defined by SWE_DISTILLER_LLM_TIMEOUT_MS, the system proceeds to the next candidate.

Sources: extensions/swe_distiller/src/extraction/llm.rs:24-65

Network & Fetch Limits

To protect against resource exhaustion and malformed web content, swe_distiller enforces strict limits on HTTP acquisition.

Operational Constraints

• Maximum Page Size: 5 MB (5 * 1024 * 1024 bytes). This limit is checked both via Content-Length headers and actual bytes received.
• Fetch Timeout: 10 seconds for standard HTTP requests.
• Retry Policy: Maximum of 3 fetch attempts with exponential backoff (300ms, 900ms, 1800ms).
• Redirect Limit: Up to 10 redirects are followed.

Sources: extensions/swe_distiller/src/fetch.rs:14-17, extensions/swe_distiller/src/fetch.rs:56-60, extensions/swe_distiller/src/fetch.rs:247-251

User-Agent Handling

The system dynamically selects User-Agents based on the target domain to optimize accessibility:

• Standard UA: Mozilla/5.0 (compatible; swe_distiller/1.0; ...)
• Bot UA: Appends bot to the string for specific domains like github.com.

Sources: extensions/swe_distiller/src/fetch.rs:19-33

Extraction Recovery Plans & Quality Gates

When heuristic extraction results in low word counts, the pipeline automatically triggers recovery plans that relax specific removal limits.

Recovery Tiers

The pipeline attempts to recover content if the word count falls below specific thresholds:

Plan	Min Word Count	Action taken on failure
Tier 1	200	Disable remove_partial_selectors.
Tier 2	50	Disable remove_hidden_elements.
Tier 3	50	Disable remove_content_patterns and remove_low_scoring.

Sources: extensions/swe_distiller/src/lib.rs:356-388

Safety & Quality Limits

LLM-generated content is subject to safety gates to ensure it has not diverged too far from the original source text:

• Word Count Ratio: The candidate must be between 75% and 135% of the original word count.
• Link Density: If the original has $\ge 6$ links, the candidate must retain at least 55% of them.
• Heading Overlap: The first heading must have at least a 0.5 (50%) token overlap with the original.
• Noise Scoring: Penalty points are applied for navigation chrome (e.g., "sign up", "search"). A candidate is rejected if its noise score is $> baseline + 2$.

Sources: extensions/swe_distiller/src/extraction/checks.rs:37-56, extensions/swe_distiller/src/extraction/llm.rs:57-60

Conclusion

Environment variables and limits in swe_distiller provide a robust framework for balancing high-quality content extraction with operational safety. By strictly limiting network input sizes and validating LLM outputs against heuristic baselines, the system prevents "hallucinations" or data loss while remaining configurable for various deployment environments through standard environment variables.