Welcome to swe-term
Relevant source files
The following files were used as context for generating this wiki page:
swe-term is a Go-based terminal/TUI Software Engineering (SWE) agent harness designed for extensibility and modularity. It provides a stable core for AI agents to interact with development environments while leveraging specialized extensions for complex tasks. The project emphasizes a "small, robust, stable core" to allow community-driven extensions without requiring deep framework knowledge.
One of the primary integrated components is swe_distiller, a Rust-based extraction pipeline that converts web URLs into clean, structured article content, primarily in Markdown. This pipeline is critical for agents that need to consume documentation or research from the web as part of their engineering workflow.
Sources: AGENTS.md:315-322, extensions/swe_distiller/ARCHITECTURE.md:3-8
System Architecture
The project is structured as a polyglot system, utilizing Go for the main terminal harness and Rust for performance-critical extraction utilities. The swe_distiller extension follows a deterministic pipeline approach to transform raw HTML into structured data.
High-Level Data Flow
The following diagram illustrates the lifecycle of a request within the swe_distiller component, from URL input to final output format.
Core Component: swe_distiller
swe_distiller is an independent Rust implementation inspired by architecture patterns that prioritize deterministic and fast extraction paths. It isolates site-specific behaviors in modular extractors and separates acquisition concerns (fetching) from extraction concerns (parsing).
Extraction Pipeline Logic
The pipeline operates through several distinct stages implemented in src/lib.rs:
- Acquisition: Fetches the page using HTTP or an optional browser fallback.
- Orchestration: Manages the flow between heuristic extractors and optional LLM providers.
- Refinement: Applies removal passes for noise (ads, navigation, boilerplate) and standardizes content structure.
- Recovery: If word counts are low, the pipeline automatically retries with relaxed extraction rules.
Sources: extensions/swe_distiller/ARCHITECTURE.md:10-14, extensions/swe_distiller/src/lib.rs:98-132, extensions/swe_distiller/src/lib.rs:259-286
Component Summary
| Component | Responsibility | Relevant Files |
|---|---|---|
| CLI Harness | Entry point for Go terminal agent interactions | main.go, extensions/swe_distiller/src/main.rs |
| Fetch Layer | Handles URL acquisition, retries, and charset decoding | fetch.rs (referenced in ARCHITECTURE.md) |
| Removal Passes | Prunes selectors, hidden elements, and clutter patterns | src/removal/, src/lib.rs:188-230 |
| LLM Orchestrator | Manages external providers like Jina or Ollama for content extraction | src/extraction/llm.rs |
| Standardizer | Normalizes footnotes, callouts, and general HTML structure | src/standardize/content.rs |
Sources: extensions/swe_distiller/ARCHITECTURE.md:27-41, extensions/swe_distiller/src/lib.rs:175-230
LLM-Enhanced Extraction
When the --llm flag is enabled, swe_distiller attempts to use configured LLM providers to improve extraction quality. The system applies safety checks before accepting LLM output, comparing it against a heuristic baseline.
LLM Configuration
LLM behavior is controlled via environment variables, allowing for pluggable providers.
| Variable | Description | Default |
|---|---|---|
SWE_DISTILLER_LLM_PROVIDERS | Comma-separated list of providers to try | gemma4:31b-cloud,reader-lm:1.5b,jina |
SWE_DISTILLER_LLM_TIMEOUT_MS | Max time for LLM response | 60000 |
SWE_DISTILLER_LLM_MAX_INPUT_CHARS | Character limit for HTML input to LLM | 60000 |
OLLAMA_BASE_URL | Endpoint for local Ollama instances | http://127.0.0.1:11434 |
Sources: extensions/swe_distiller/src/extraction/llm.rs:12-14, extensions/swe_distiller/README.md:30-36
Operational Guidelines for Agents
The harness is designed for AI-agent extensibility. Agents interacting with this system should follow the "Polyglot Principal Engineer" persona, prioritizing correctness, safety, and observability.
Decision Hierarchy for System Modifications
- Correctness & User Intent: Do what was asked.
- Safety & Blast Radius: Protect secrets and production systems.
- Observability: Failures must be visible and diagnosis cheap.
- Operational Simplicity: Ensure ease of deployment and rollbacks.
Sources: AGENTS.md:30-48, AGENTS.md:315-322
swe-term serves as a robust platform for software engineering automation, combining a Go-based TUI for interaction with advanced Rust-based content extraction, all governed by strict architectural principles to ensure maintainability and agent compatibility.