Project Roadmap
Relevant source files
The following files were used as context for generating this wiki page:
The Project Roadmap outlines the strategic evolution and technical milestones for the swe-term repository, with a primary focus on the swe_distiller extension. swe_distiller serves as a Rust-first extraction pipeline designed to transform raw URLs into structured Markdown content for consumption by AI agents.
The roadmap is driven by principles of correctness, safety, and maintainability, ensuring that the tool remains a robust utility for automated software engineering tasks. Future developments prioritize expanding content type support, hardening acquisition layers, and maturing the integration of Large Language Models (LLMs) for high-fidelity content extraction.
Sources: extensions/swe_distiller/ARCHITECTURE.md:3-8, AGENTS.md:33-45, extensions/swe_distiller/README.md:3-5
1. Core Development Pillars
The project roadmap is structured around seven key technical domains. These focus on transitioning from a purely HTML-based pipeline to a generalized multi-content acquisition system.
Planned Technical Milestones
| Category | Priority Goal | Key Components |
|---|---|---|
| Multi-Content Pipeline | Support non-HTML sources | InputKind (PDF/Image), RawDocument contracts |
| Browser Hardening | Improve stealth and reliability | Challenge detection markers, binary prerequisite docs |
| LLM Pipeline Maturity | Quantify and stabilize LLM output | Provider-level metrics, explicit error categories |
| Extractor Coverage | Enhance site-specific accuracy | YouTube transcripts, Reddit/X thread stitching |
| Markdown Quality | Improve AST conversion | Nested list indentation, rich table handling |
| Observability | Standardize diagnostics | Stable event schema, automated agent diagnostics |
| Packaging/UX | Streamline developer experience | Shell completion, release profile binary size checks |
Sources: extensions/swe_distiller/TODO.md:1-51
2. Multi-Content Acquisition Strategy
A significant portion of the roadmap involves abstracting the current fetch layer to support multiple document types. This involves introducing explicit acquisition contracts that route documents by their InputKind before parsing begins.
This flow visualizes the planned transition from a decoded-HTML-only pipeline to a routed document processing system.
Sources: extensions/swe_distiller/TODO.md:3-16, extensions/swe_distiller/ARCHITECTURE.md:86-93
3. LLM Integration Roadmap
The project intends to transition the LLM extraction path from an experimental feature to a primary, verified extraction mode. Currently, the pipeline tries configured providers in order and applies safety checks before accepting output.
Planned Improvements for LLM Pipeline
- Metric Tracking: Addition of latency and rejection reason metrics in debug logs.
- Error Classification: Implementing specific categories for timeout, safety rejections, and malformed responses.
- Fallback Reliability: Hardening the heuristic fallback when LLM providers fail or produce low-quality results.
Sources: extensions/swe_distiller/TODO.md:21-25, extensions/swe_distiller/src/extraction/llm.rs:40-66, extensions/swe_distiller/src/lib.rs:37-65
4. Extractor and Conversion Maturity
The roadmap identifies specific areas where extraction accuracy needs improvement to handle complex web structures, particularly for developer-centric and social media platforms.
Target Improvements
- Social & Media: Enhancing Hacker News comment tree shaping, YouTube transcript handling, and X/Twitter thread stitching.
- AST Conversion: Improving the
markdown_ast.rslogic to handle uneven table row widths and stronger code language inference from CSS classes. - Sanitization: Hardening the
standardize/content.rslogic against malformed HTML edge cases to prevent conversion failures.
Sources: extensions/swe_distiller/TODO.md:27-41, extensions/swe_distiller/src/markdown_ast.rs:172-205, extensions/swe_distiller/src/standardize/content.rs:20-33
5. Architectural Evolution
The architecture is designed to evolve from a modular monolith towards a service-oriented structure as technical pain dictates. The roadmap emphasizes maintaining a small, robust core for the swe-term harness while allowing extensions like swe_distiller to operate as isolated Rust components.
The diagram above represents the current high-level orchestration which is scheduled for expansion with multi-content kind classification.
Sources: extensions/swe_distiller/ARCHITECTURE.md:22-42, extensions/swe_distiller/src/lib.rs:24-65, AGENTS.md:219-225
Summary of Significant Milestone Items
The Project Roadmap focuses on transforming swe_distiller into a multi-modal document processor. By separating acquisition concerns from extraction logic and implementing standardized observability, the project aims to provide a deterministic, high-fidelity data source for AI-driven software engineering tools within the swe-term ecosystem.
Sources: extensions/swe_distiller/TODO.md:1-51, extensions/swe_distiller/ARCHITECTURE.md:10-20