Frugal Engineering & Philosophy

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• extensions/swe_distiller/ARCHITECTURE.md
• extensions/swe_distiller/src/lib.rs
• extensions/swe_distiller/src/extraction/llm.rs
• extensions/swe_distiller/src/extraction/checks.rs
• extensions/swe_distiller/src/fetch.rs

Frugal Engineering within the swe-term project represents a disciplined approach to software architecture that prioritizes minimal complexity, maximum leverage, and operational simplicity. The project adheres to a "Unix-native" mindset, favoring small, composable tools over monolithic frameworks and choosing technologies that minimize environmental lock-in. This philosophy is most evident in the development of the swe_distiller extension, which emphasizes fast, deterministic heuristic paths before escalating to more resource-intensive solutions like LLMs or full browser automation.

The core mission is to solve problems with elegant simplicity, treating code as a cost rather than an asset. The project follows a strict decision hierarchy that values correctness, safety, and observability above performance heroics or aesthetic elegance. By starting with the simplest viable implementation (e.g., shell scripts or modular monoliths) and only moving toward complexity when necessitated by measurable pain, the system maintains a high degree of maintainability and portability.
Sources: AGENTS.md:14-19, AGENTS.md:209-222, extensions/swe_distiller/ARCHITECTURE.md:7-11

Core Engineering Principles

The project's architectural decisions are guided by a specific hierarchy of values. When principles conflict, the system prioritizes correctness and safety over maintainability and performance.

Decision Hierarchy

Priority	Principle	Description
1	Correctness & Intent	Fulfilling the specific user request accurately.
2	Safety & Blast Radius	Protecting data and secrets; avoiding irreversible damage.
3	Observability	Ensuring failures are visible and diagnosis is inexpensive.
4	Operational Simplicity	Systems must be operable by humans under pressure (e.g., 3 AM).
5	Maintainability	Code must be readable and testable six months later.
6	Portability	Avoiding unnecessary environment or vendor lock-in.
7	Performance	Optimize only when required; measure before acting.

Sources: AGENTS.md:24-42

The Rule of Three and Evolutionary Paths

The project avoids premature abstraction by following the Rule of Three: implement twice, notice duplication, and only abstract on the third occurrence. Architecturally, components are encouraged to evolve from left to right along the following path, moving only when forced by operational pain:
Monolith → Modular Monolith → Services → Microservices
Sources: AGENTS.md:231-236, AGENTS.md:239-243

Technical Implementation: The Distillation Pipeline

The swe_distiller component embodies frugal engineering by utilizing a multi-stage pipeline that favors deterministic heuristics. It attempts to extract content using the cheapest methods first (HTML parsing and site-specific extractors) before optionally engaging LLMs.

Heuristic-First Extraction

The pipeline is designed to be "Rust-first" and deterministic. It separates acquisition (fetching) from extraction logic to ensure that the core path remains fast.
Sources: extensions/swe_distiller/ARCHITECTURE.md:7-19

Recovery and Relaxation Plans

Frugal engineering is further demonstrated through "Recovery Plans." If the initial extraction yields insufficient content (e.g., low word count), the system automatically retries by relaxing pruning constraints, such as disabling partial selector removal or hidden element pruning.
Sources: extensions/swe_distiller/src/lib.rs:252-277

LLM Strategy: Conditional Escalation

The use of Large Language Models (LLMs) is treated as a high-leverage but resource-intensive fallback. The swe_distiller implementation uses a "provider ordering" strategy, attempting multiple providers (e.g., Ollama, Jina) only if configured.

Safety and Noise Checks

To prevent LLM hallucinations or low-quality output from replacing deterministic results, the system employs strict safety checks. A candidate LLM output is only accepted if it passes word count, link count, and heading overlap comparisons against the baseline heuristic output.
Sources: extensions/swe_distiller/src/extraction/llm.rs:36-62, extensions/swe_distiller/src/extraction/checks.rs:38-58

Operational Discipline

The project mandates a "Systems Preflight" before execution, emphasizing the assessment of "blast radius" and reversibility. This is particularly critical when dealing with production systems or destructive actions.

• Idempotency: All scripts and execution tasks must be idempotent to ensure safety during retries.
• Observability: Structured debug logging is used to track pipeline deltas, word count changes per removal step, and LLM outcomes.
• Tool Selection: The project explicitly prefers structured AST-based tools (like ast-grep) over regex for non-trivial code modifications, recognizing that "Source code has structure; regex does not."
  Sources: AGENTS.md:52-65, AGENTS.md:188-193, AGENTS.md:167-172, extensions/swe_distiller/ARCHITECTURE.md:73-80

Secrets Handling

Frugal security practices include a strict dependency on the Doppler CLI for secret management. Secrets are never retrieved as raw values in logs or prompts; instead, they are injected into the runtime environment via doppler run.
Sources: AGENTS.md:267-285

Conclusion

Frugal Engineering & Philosophy in swe-term is not merely about cost-cutting, but about minimizing the "surface area" of failure. By prioritizing simple, deterministic tools and establishing clear gates for complex escalations, the project ensures that the most reliable component—the one that does not exist—is favored whenever possible.
Sources: AGENTS.md:260-264, extensions/swe_distiller/ARCHITECTURE.md:7-11