Markdown AST Conversion
Relevant source files
The following files were used as context for generating this wiki page:
Markdown AST Conversion is a centralized, Rust-first extraction stage within swe_distiller that transforms cleaned HTML into structured Markdown content. It serves as the final structural phase of the pipeline, converting the DOM (Document Object Model) into a readable format while applying deterministic cleanup rules, structural normalization, and post-processing.
Sources: extensions/swe_distiller/ARCHITECTURE.md:1-25, extensions/swe_distiller/src/lib.rs:200-220
Architecture and Pipeline Integration
The conversion process is the final step in the swe_distiller pipeline. Before conversion, the document undergoes acquisition, extractor selection, and various cleanup passes (removing hidden elements, small images, or low-scoring blocks). The conversion is "AST-based," meaning it traverses the parsed HTML tree using the scraper and ego-tree crates to generate Markdown equivalents.
The pipeline supports two primary execution paths: a fast, deterministic heuristic path and an optional LLM extraction override (--llm). The AST conversion is used as a baseline even in LLM mode to verify safety and quality gates.
Sources: extensions/swe_distiller/ARCHITECTURE.md:20-35, extensions/swe_distiller/src/extraction/llm.rs:26-35, extensions/swe_distiller/src/lib.rs:200-250
AST Conversion Logic
The core conversion resides in markdown_ast.rs. It utilizes a recursive DOM-walking strategy to translate HTML tags into Markdown syntax.
The Render Context
A RenderCtx struct tracks state during the traversal, specifically the list_depth to ensure correct indentation for nested ordered and unordered lists.
Sources: extensions/swe_distiller/src/markdown_ast.rs:7-10
Element Mapping
The converter maps standard HTML elements to their Markdown counterparts using the render_element function.
| HTML Tag | Markdown Transformation | Notes |
|---|---|---|
h1-h6 | # repetition (level 1-6) | Headings are trimmed and wrapped in newlines. |
p | Standard block text | Paragraphs are separated by double newlines. |
a | [text](https://github.com/cercova-studios/swe-term/blob/f6e0056974bfecb921b64e8829e3b03cc29c1906/href) | Skips anchor permalinks containing only punctuation. |
pre | ```lang\ncode\n``` | Attempts language detection via class attributes. |
blockquote | > quote | Recursively handles multi-line quotes. |
ul / ol | - or 1. | Handles nested lists using list_depth indentation. |
table | GFM Table format | Converts rows to ` |
hr | --- | Horizontal rules. |
br | \n | Manual line breaks. |
Sources: extensions/swe_distiller/src/markdown_ast.rs:45-125
Recursive Traversal Sequence
Structural Post-Processing
After the AST traversal, the generated string is passed to markdown.rs for refinement. This stage fixes artifacts introduced by raw HTML-to-text conversion.
Normalization and Cleanup
The postprocess_markdown function executes several regex-based passes:
- Metadata Run-ons: Splits text where links or dates are immediately followed by sentences without sufficient whitespace.
- Chrome Removal: Strips known UI elements that extractors might have missed, such as "Sign up," "Share this article," or "More blog posts to read."
- Bold Span Correction: Fixes instances where bold markers (
**) cross heading boundaries, which often breaks Markdown rendering. - Newline Normalization: Collapses multiple newlines (3+) into double newlines for consistent spacing.
Sources: extensions/swe_distiller/src/markdown.rs:163-170, extensions/swe_distiller/src/markdown.rs:200-240
Heading Management
To ensure every distilled article has a proper entry point, the system applies two specific title-related functions:
guess_title: Identifies the first#heading in the Markdown.ensure_title_heading: If no H1 heading is found, or if the title metadata doesn't appear early in the document, it injects a# Titleheading at the very start.
Sources: extensions/swe_distiller/src/markdown.rs:136-161
Implementation Details
Language Detection
The converter attempts to infer the programming language for code blocks by scanning class attributes on <pre> and <code> tags for patterns like language-rust or highlight-source-js.
// extensions/swe_distiller/src/markdown_ast.rs:248-268
fn detect_code_language(el: &ElementRef<'_>) -> Option<String> {
static LANG_RE: Lazy<Regex> = Lazy::new(|| {
Regex::new(r"(? :^|\s)(? :language|lang|highlight-source)-([a-zA-Z0-9_+-]+)(? :\s|$)")
.expect("valid regex")
});
// ... logic to check attributes ...
}
Table Conversion
Tables are converted to Markdown by selecting all tr elements and extracting text from th (header) or td (data) cells. It automatically generates the separator row (| --- |) required for GitHub Flavored Markdown.
Sources: extensions/swe_distiller/src/markdown_ast.rs:205-235
Summary
Markdown AST Conversion in swe_distiller provides a robust mechanism for turning processed HTML into clean, standardized Markdown. By combining structural DOM traversal with aggressive post-processing normalization, it ensures that the output is not only semantically correct but also optimized for readability and compatibility with various Markdown renderers.
Sources: extensions/swe_distiller/ARCHITECTURE.md, extensions/swe_distiller/src/lib.rs