Extractor Service

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/extractor/init.py
• monitor/extractor/article_parser.py
• monitor/extractor/metadata.py
• monitor/models/article.py
• monitor/main.py

The Extractor Service is a core component of the Technical Blog Monitor responsible for parsing raw HTML content from web pages. Its primary function is to extract the main article text, associated metadata (such as author, publication date, and tags), and media links. It uses a combination of readability-lxml for primary content extraction and BeautifulSoup for more granular parsing, cleaning, and metadata discovery.

This service acts as a bridge between the raw, unstructured web content fetched by other parts of the system and the structured, clean data required for the embedding and vector storage stages. It is designed to be robust, employing several fallback strategies to handle diverse and often messy HTML structures found across different technical blogs. All extracted information is encapsulated within the ArticleContent data model.

Sources: monitor/extractor/init.py, monitor/extractor/article_parser.py:10-16

Core Extraction Flow

The main entry point for the extraction process is the extract_article_content function. This orchestrator manages caching, HTML fetching (if necessary), and coordinates the various parsing and cleaning sub-routines. The process is designed to be efficient by offloading CPU-bound parsing tasks to a thread pool, preventing the main asyncio event loop from being blocked.

Sources: monitor/extractor/article_parser.py:91-107

The following diagram illustrates the high-level data flow within the extract_article_content function.

Data Model: ArticleContent

All extracted data is structured into a Pydantic ArticleContent model. This model serves as the standardized data transfer object for a processed article throughout the application pipeline.

Sources: monitor/models/article.py

Key Fields

The table below details the most important fields in the ArticleContent model.

Field	Type	Description
url	HttpUrl	The canonical URL of the article.
title	str	The main title of the article.
text	str	The cleaned, plain-text version of the article content.
html	str	The cleaned HTML of the main article body.
author	Optional[str]	The extracted author's name.
publish_date	Optional[datetime]	The publication date of the article, timezone-aware.
summary	Optional[str]	A short summary or description of the article.
word_count	int	The total number of words in the cleaned text.
image_urls	List[str]	A list of absolute URLs for images found in the article.
screenshot_paths	List[str]	Local filesystem paths to screenshots of the rendered article.
tags	List[str]	A list of normalized tags or keywords.
content_type	ContentType	An enum classifying the content (e.g., ARTICLE, TUTORIAL).
reading_time_minutes	Optional[float]	An estimated reading time, calculated automatically from word_count.

Sources: monitor/models/article.py:18-43, monitor/models/article.py:80-86

Class Diagram

This diagram shows the structure of the ArticleContent model and its relationship with the ContentType enum.

Content & Text Extraction

The service employs a multi-stage process to extract clean text from potentially noisy HTML.

Primary Extraction with Readability

The primary method for extracting the main article body is readability-lxml. The extract_main_content function wraps readability.Document to get the article's title and a summarized HTML containing only the core content. This is executed in a thread pool to avoid blocking I/O operations.

Sources: monitor/extractor/article_parser.py:216-240

Fallback with BeautifulSoup

If readability fails, the system falls back to extract_content_with_soup. This function uses BeautifulSoup to parse the HTML and attempts to find the main content by searching for common container selectors. If a suitable container is not found, it defaults to using the entire <body> tag.

Sources: monitor/extractor/article_parser.py:242-258, monitor/extractor/article_parser.py:261-292

Text Cleaning

Once the main HTML content is isolated, the clean_article_text function performs several cleaning steps:

1. It removes a predefined list of "noise" elements.
2. It strips all HTML comments.
3. It extracts text from the remaining elements, intelligently adding newlines for block-level tags like <p> and <h1>.
4. It normalizes whitespace by collapsing multiple spaces and newlines.

The following elements are considered noise and are removed:
script, style, iframe, form, button, input, nav, footer, header, aside, noscript, figcaption, figure, time, svg.

Sources: monitor/extractor/article_parser.py:295-336, monitor/extractor/article_parser.py:46-49

Metadata Extraction

Metadata extraction is a critical and complex part of the service, handled by functions in both article_parser.py and metadata.py. It uses a hierarchical approach, trying multiple sources for each piece of data to ensure the highest quality result.

Sources: monitor/extractor/metadata.py, monitor/extractor/article_parser.py:371-440

Metadata Search Strategy

For each metadata field (like author or date), the extractor queries sources in a specific order of preference.

Author Extraction

The extract_author function searches for the author's name in the following locations:

1. Meta tags: <meta name="author">, <meta name="dc.creator">, <meta property="article:author">
2. Open Graph: <meta property="og:author">
3. Schema.org: Elements with itemprop="author"
4. Common CSS selectors: .author, .byline, [rel="author"]
5. Regex patterns (e.g., by:, written by:) as a last resort.

Sources: monitor/extractor/metadata.py:120-151

Publication Date Extraction

The extract_publish_date function attempts to find and parse a date string from:

1. Meta tags: <meta name="date">, <meta name="pubdate">, <meta property="article:published_time">
2. Open Graph: <meta property="og:published_time">
3. Schema.org: Elements with itemprop="datePublished"
4. <time> elements with a datetime attribute.
5. Common CSS selectors: .date, .published, .post-date
6. Regex patterns for formats like YYYY-MM-DD and Month DD, YYYY.

All found date strings are parsed using dateutil.parser.

Sources: monitor/extractor/metadata.py:173-223

Tag Extraction

The extract_tags function aggregates keywords from:

1. Meta tags: <meta name="keywords">
2. Open Graph: <meta property="article:tag">
3. Common CSS selectors for tag containers: .tags, .categories, .keywords

All found tags are collected, deduplicated, and normalized to lowercase.

Sources: monitor/extractor/metadata.py:226-261

Integration in the Application

The Extractor Service is invoked within the process_post function in the main application module. This function orchestrates the entire pipeline for a single blog post.

The sequence diagram below shows how extract_article_content fits into the process_post workflow.

Conclusion

The Extractor Service is a sophisticated and resilient component designed to handle the complexities of parsing real-world web content. By combining specialized libraries, implementing a layered search strategy with fallbacks, and integrating cleanly with the application's asynchronous architecture, it reliably transforms messy HTML into structured, valuable data ready for analysis and semantic search.