Testing Strategy & Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
- extensions/swe_distiller/tests/integration.rs
- extensions/swe_distiller/tests/fixtures/general--cp4space-jordan-algebra.html
- extensions/swe_distiller/src/extraction/checks.rs
- extensions/swe_distiller/src/markdown.rs
- extensions/swe_distiller/src/markdown_ast.rs
- extensions/swe_distiller/src/lib.rs
- extensions/swe_distiller/src/extraction/llm.rs
- extensions/swe_distiller/ARCHITECTURE.md
The testing strategy for swe_distiller is designed to ensure the accuracy, safety, and reliability of the web content extraction pipeline. It employs a multi-layered approach ranging from unit tests for specific heuristic checks to comprehensive integration tests using real-world HTML fixtures. This ensures that the transition from raw HTML to cleaned Markdown maintains structural integrity while effectively stripping "chrome" (navigation, footers, and advertisements).
The strategy specifically addresses two primary extraction paths: the deterministic heuristic pipeline and the optional LLM-based extraction pipeline. By utilizing safety gates and quality scoring, the system verifies that automated extractions do not lose critical information or introduce excessive noise.
Integration Testing Framework
Integration tests verify the end-to-end flow of the parse_html function, ensuring that different site structures (e.g., Substack, GitHub, X/Twitter) are handled correctly by their respective extractors. These tests often use include_str! to load complex HTML documents from a dedicated fixtures directory.
Key Integration Test Patterns
- Smoke Tests: Validating that large, real-world fixtures produce non-empty output with significant word counts.
- Chrome Removal: Asserting that specific UI elements like "Contact Sales" or "Sign Up" are successfully removed from the output.
- Metadata Verification: Ensuring titles, authors, and publication dates are correctly extracted from meta tags or JSON-LD preloads.
- Structural Integrity: Checking that paragraph breaks, headings, and lists are preserved during the conversion to Markdown.
Sources: extensions/swe_distiller/tests/integration.rs:37-47, extensions/swe_distiller/tests/integration.rs:50-86, extensions/swe_distiller/ARCHITECTURE.md:21-30
Logic Flow for Extraction Testing
The following diagram illustrates how an integration test validates the pipeline from HTML input to the final DistillerResponse.
The diagram shows the sequence of operations within a standard integration test to verify a successful extraction.
Sources: extensions/swe_distiller/src/lib.rs:72-74, extensions/swe_distiller/tests/integration.rs:25-35
LLM Safety & Quality Testing
When the --llm mode is enabled, the system uses a specific suite of safety checks to prevent hallucinations or excessive data loss. These checks compare the LLM-generated Markdown against a baseline produced by the heuristic pipeline.
Safety Checks and Quality Gates
Testing for LLM output involves two primary metrics:
- Word Count & Link Ratios: Validating that the candidate text isn't significantly shorter or longer than the original (e.g., between 75% and 135% of the original word count).
- Noise Scoring: A heuristic-based score that penalizes navigation chrome, repeated single characters, and short lines that lack heading markers.
Sources: extensions/swe_distiller/src/extraction/checks.rs:43-69, extensions/swe_distiller/src/extraction/llm.rs:46-56
Quality Scoring Parameters
| Check Type | Logic / Threshold | Purpose |
|---|---|---|
word_count | cw < ow * 0.75 or cw > ow * 1.35 | Prevents extreme data loss or hallucinated padding. |
link_count | cl < ol * 0.55 (if ol >= 6.0) | Ensures essential reference links are preserved. |
noise_score | Penalizes "Sign Up", "Search", digit-only lines | Detects if extraction failed to strip UI chrome. |
heading_overlap | < 0.5 shared tokens in H1 | Ensures the topic of the article hasn't changed. |
Sources: extensions/swe_distiller/src/extraction/checks.rs:1-30, extensions/swe_distiller/src/extraction/checks.rs:43-69
Markdown Conversion Accuracy
Tests within markdown.rs and markdown_ast.rs focus on the technical details of the Markdown AST converter. These ensure that complex HTML elements like tables, nested lists, and code blocks are formatted correctly.
Post-processing Logic
The testing suite verifies that the postprocess_markdown function handles common "run-on" errors where metadata (like dates) is accidentally merged with article sentences.
This diagram shows the relationship between the AST conversion and the post-processing steps that ensure clean final output.
Sources: extensions/swe_distiller/src/markdown_ast.rs:13-28, extensions/swe_distiller/src/markdown.rs:202-208
Key AST Rendering Features
- Code Language Detection: Extracting language hints from CSS classes (e.g.,
language-rust). - Table Formatting: Converting
<table>,<tr>,<th>, and<td>into standard Markdown pipe tables. - Nested Lists: Maintaining indentation for
<ul>and<ol>tags.
Sources: extensions/swe_distiller/src/markdown_ast.rs:136-170, extensions/swe_distiller/src/markdown_ast.rs:253-273
Fixtures and Mock Data
The project utilizes local HTML files to simulate various site structures. This allows for regression testing against specific platforms without requiring live network access during the build process.
general--cp4space-jordan-algebra.html: A fixture representing a technical blog post with deep mathematical content and standard blog chrome.- Inline HTML strings: Used for targeted unit tests to verify specific regex patterns or DOM edge cases.
Sources: extensions/swe_distiller/tests/integration.rs:38, extensions/swe_distiller/src/markdown.rs:278-300
Conclusion
The testing strategy for swe_distiller prioritizes correctness and safety. By combining deterministic heuristic assertions with LLM safety gates, the project ensures that extracted content is both readable and factually representative of the source URL. The heavy use of fixtures and AST-based verification provides a robust framework for expanding extractor coverage across the web.