Feature: Content Extraction
Relevant source files
The following files were used as context for generating this wiki page:
The Content Extraction feature is a core component of the Technical Blog Monitor, responsible for converting raw HTML from a blog post URL into a structured, clean, and metadata-rich format. It fetches the web page, parses the main article body, extracts key metadata like author and publication date, identifies images, and cleans the text content. The final output is a Pydantic model, ArticleContent, which is used for downstream processing such as embedding generation and storage.
This system is designed to be robust, employing a primary extraction engine (readability-lxml) with a fallback mechanism (BeautifulSoup) and multiple strategies for finding metadata. CPU-bound parsing tasks are offloaded to a thread pool to avoid blocking the main async event loop. The entire process is orchestrated by the extract_article_content function.
Sources: monitor/extractor/article_parser.py:15-22, monitor/extractor/init.py, WARP.md
Core Orchestration
The primary entry point for content extraction is the extract_article_content async function. It manages the entire lifecycle of fetching, parsing, and structuring article data.
Sources: monitor/extractor/article_parser.py:68-89
High-Level Data Flow
The process begins with a URL and orchestrates caching, fetching, and parsing operations. CPU-intensive tasks like HTML parsing and text cleaning are executed in a ThreadPoolExecutor to prevent blocking the asynchronous event loop.
The following diagram illustrates the high-level workflow of the extract_article_content function.
This diagram shows the control flow within extract_article_content, including caching, HTML fetching, and concurrent processing steps.
Sequence Diagram
This sequence diagram details the interactions between the caller, the main function, the cache, and the thread pool.
This diagram illustrates the sequence of operations, highlighting the use of a cache and a thread pool for performance.
Content and Text Processing
Main Content Extraction
The system first attempts to extract the main article content using the readability-lxml library. This is performed within a thread pool to avoid blocking.
If readability-lxml fails, it falls back to a custom BeautifulSoup-based function, extract_content_with_soup. This fallback function attempts to find the main content by iterating through a list of common CSS selectors.
Sources: monitor/extractor/article_parser.py:203-242, monitor/extractor/article_parser.py:245-283
| Component | Description | Source File |
|---|---|---|
| Primary Extractor | readability.Document | monitor/extractor/article_parser.py:215-218 |
| Fallback Extractor | extract_content_with_soup | monitor/extractor/article_parser.py:230-233 |
The fallback mechanism uses the following selectors to identify the main article container:
| Type | Selectors |
|---|---|
| Article Selectors | article, main, .post, .article, .entry, .content, .post-content, .entry-content, .article-content, .blog-post, .blog-entry, #content, #main, #post, #article |
Sources: monitor/extractor/article_parser.py:59-63
Text Cleaning
Once the main HTML content is extracted, the clean_article_text function sanitizes it to produce clean, readable plain text. This process involves:
- Parsing the HTML with
BeautifulSoup. - Removing a predefined list of "noise" elements.
- Extracting text from remaining elements, adding newlines for block-level tags like
<p>and<h2>to preserve structure. - Normalizing whitespace by collapsing multiple spaces and newlines.
Sources: monitor/extractor/article_parser.py:286-326
The following HTML tags are considered noise and are removed during cleaning:
| Type | Tags |
|---|---|
| Noise Elements | script, style, iframe, form, button, input, nav, footer, header, aside, noscript, figcaption, figure, time, svg |
Sources: monitor/extractor/article_parser.py:52-56
Metadata Extraction
A significant part of the extraction process is gathering rich metadata from the article. This is handled by extract_article_metadata (async wrapper) and extract_metadata_sync (synchronous implementation). The system uses a multi-layered approach to find each piece of data.
Sources: monitor/extractor/article_parser.py:329-423, monitor/extractor/metadata.py:40-117
The following diagram shows the hierarchical strategy for finding metadata.
This diagram shows that for each metadata field (like Author or Date), the system tries multiple sources in a specific order.
Metadata Extraction Strategies
The system employs a prioritized search for each metadata type.
| Metadata Type | Extraction Sources (in order of priority) | Source Function |
|---|---|---|
| Author | 1. <meta> tags (author, dc.creator). 2. Open Graph (og:author). 3. Schema.org (itemprop="author"). 4. Common CSS selectors (.author, .byline). 5. Regex patterns (by:, author:). | extract_author |
| Publish Date | 1. <meta> tags (pubdate, article:published_time). 2. Open Graph (og:published_time). 3. Schema.org (itemprop="datePublished"). 4. <time> element. 5. Common CSS selectors (.date, .published). 6. Regex patterns. | extract_publish_date |
| Tags | 1. <meta name="keywords">. 2. Open Graph (article:tag). 3. Common CSS selectors (.tags, .categories). | extract_tags |
| Summary | 1. <meta name="description">. 2. Open Graph (og:description). 3. Twitter Card (twitter:description). 4. Schema.org (itemprop="description"). 5. Common CSS selectors (.summary, .excerpt). | extract_summary / extract_description |
Sources: monitor/extractor/article_parser.py:426-673, monitor/extractor/metadata.py:120-307
Image Extraction
The extract_image_urls function is responsible for finding all relevant images within the extracted article HTML. Its logic includes:
- Finding all
<img>tags and reading theirsrcordata-srcattributes. - Parsing
styleattributes to findbackground-imageURLs. - Resolving relative URLs to absolute URLs using the article's base URL.
- Filtering out small images (less than 100x100 pixels) if dimensions are available.
- Removing duplicate URLs.
Sources: monitor/extractor/article_parser.py:901-948
Data Model: ArticleContent
The final output of the extraction process is an instance of the ArticleContent Pydantic model. This model serves as the standardized data structure for a processed article.
Sources: monitor/models/article.py
Class Diagram
This diagram shows the structure of the ArticleContent model and its fields.
Field Descriptions
| Field | Type | Description |
|---|---|---|
url | HttpUrl | The canonical URL of the article. |
title | str | The main title of the article. |
text | str | The cleaned, plain text content. |
html | str | The cleaned HTML of the main article body. |
author | Optional[str] | The extracted author's name. |
publish_date | Optional[datetime] | The publication date, timezone-aware. |
summary | Optional[str] | A short summary or description. |
word_count | int | The total number of words in the text. |
image_urls | List[str] | A list of absolute URLs for images found in the article. |
tags | List[str] | A list of normalized tags or keywords. |
metadata | Dict[str, Any] | A dictionary containing all raw extracted metadata (including OG, Twitter, etc.). |
content_type | ContentType | An enum classifying the article type (e.g., BLOG_POST, TUTORIAL). |
reading_time_minutes | Optional[float] | Calculated estimated reading time. |
extracted_at | datetime | Timestamp of when the extraction was performed. |
Sources: monitor/models/article.py:19-45, monitor/models/article.py:68-74
Content Type Determination
The determine_content_type function attempts to classify the article into a specific category using heuristics. It checks for clues in the metadata and title.
- Metadata Clues: Checks
og_typeandjsonld_typefor explicit types likearticle,BlogPosting, orTechArticle. - Title Keywords: Scans the lowercased title for keywords like
tutorial,guide,release,documentation, orcase study. - URL Clues: Checks if "blog" is present in the URL.
- Default: If no other clues are found, it defaults to
ContentType.BLOG_POST.
Sources: monitor/extractor/article_parser.py:951-989
Configuration
While most extraction logic is self-contained, some behaviors can be influenced by the main application configuration defined in monitor/config.py.
| Configuration Section | Key | Description |
|---|---|---|
ArticleProcessingConfig | full_content_capture | If False, the entire extraction process described here is skipped, and only feed metadata is stored. |
ArticleProcessingConfig | archive_html | If True, the cleaned article HTML is persisted. |
ArticleProcessingConfig | archive_screenshots | If True, screenshots taken during rendering are persisted. |
Sources: monitor/config.py:221-268
Summary
The Content Extraction feature is a sophisticated and resilient module designed to handle the complexities of modern web pages. By combining readability-lxml, BeautifulSoup, and a multi-layered metadata search strategy, it reliably transforms unstructured HTML into the clean, structured ArticleContent model. Its use of a thread pool for heavy computation ensures that it remains performant within the application's asynchronous architecture. This clean data is fundamental for all subsequent stages in the monitoring pipeline, including embedding and semantic search.