Fetcher Service
Relevant source files
The following files were used as context for generating this wiki page:The Fetcher Service is a core component of the Technical Blog Monitor responsible for all content retrieval from the web. It provides a dual-mode approach to fetching: a lightweight, fast HTTP client for simple requests like RSS/Atom feeds, and a powerful, full-featured headless browser for rendering JavaScript-heavy pages and capturing rich content like screenshots. This separation allows the system to use the most efficient method for each task.
The service is designed for high-concurrency and resilience, incorporating features like connection pooling, retry mechanisms with exponential backoff, and robust resource management. Its components are exposed through the monitor.fetcher package.
Sources: monitor/fetcher/init.py, README.md
Architecture and Components
The Fetcher Service is composed of two primary, independent components: a simple asynchronous HTTP client and a pool of headless browser instances managed by Playwright.
This diagram shows the two main components of the Fetcher Service and their typical usage within the application.
Sources: monitor/fetcher/init.py, monitor/feeds/base.py:180-244, monitor/main.py:255-285
HTTP Client
The HTTP client functionality is primarily used for fetching structured data feeds like RSS, Atom, or JSON. It is built around httpx.AsyncClient for high-performance, asynchronous I/O.
A key function is fetch_with_retry, which wraps HTTP GET requests with a resilient retry mechanism using the tenacity library. This handles transient network errors and server-side issues gracefully.
The diagram above illustrates the retry logic of the fetch_with_retry function.
Sources: monitor/feeds/base.py:110-141, monitor/fetcher/http_client.py
This functionality is not encapsulated in a dedicated AsyncHTTPClient class but is implemented as standalone functions within the feeds and fetcher modules. Feed processors like RSSFeedProcessor use an httpx.AsyncClient directly to perform requests.
Sources: monitor/feeds/rss.py:46-70
Browser Rendering (BrowserPool)
For rendering full article pages, especially those that rely heavily on JavaScript, the system uses the BrowserPool. This component manages a pool of concurrent Playwright browser instances to render pages, extract fully-formed HTML, and take screenshots.
Class Structure
The BrowserPool acts as an asynchronous context manager, handling the lifecycle of browser instances. It ensures that browsers are launched on entry and gracefully closed on exit. The BrowserContext class manages a single browser instance within the pool.
This diagram shows the relationship between the main application context, the browser pool, and individual browser contexts.
Sources: monitor/fetcher/browser.py:28-184, monitor/main.py:50-51, 98-104
Rendering and Screenshot Flow
The primary method is render_and_screenshot. When called, it acquires a browser from the pool, creates a new page, navigates to the URL, and takes a screenshot. It supports features like ad blocking and stealth mode to improve content extraction quality.
This sequence diagram details the steps involved in rendering a page and capturing a screenshot.
Sources: monitor/fetcher/browser.py:141-184, monitor/feeds/base.py:410-436
Configuration
The browser-based fetcher is configured via the BrowserConfig model. These settings allow fine-tuning of performance, resource usage, and rendering behavior.
| Setting | Type | Default | Description |
|---|---|---|---|
headless | bool | True | Runs the browser in headless mode without a visible UI. |
browser_type | str | "chromium" | The browser engine to use (chromium, firefox, webkit). |
timeout_seconds | int | 30 | Timeout for page navigation and rendering. |
max_concurrent_browsers | int | 3 | The maximum number of browser instances to run concurrently. |
block_ads | bool | True | Enables ad-blocking to clean up page content before extraction. |
stealth_mode | bool | True | Applies evasions to make the headless browser appear more like a regular user. |
screenshot_full_page | bool | True | Captures the entire scrollable page, not just the viewport. |
screenshot_dir | Path | data/screenshots | Directory to store captured screenshots. |
wait_until | str | "networkidle" | The page loading state to wait for before considering navigation complete. |
Sources: monitor/config.py:40-66
Usage in the Application
The fetcher components are integrated at different stages of the content processing pipeline.
Feed Discovery
During feed discovery, an httpx.AsyncClient is used within each FeedProcessor (e.g., RSSFeedProcessor) to fetch the raw feed file (XML or JSON). This is a lightweight operation that does not require a full browser. The fetch_feed method in the processor handles this.
Sources: monitor/feeds/rss.py:46-99, monitor/feeds/base.py:180-244
Full Article Processing
Once new posts are discovered from a feed, the BrowserPool is used to process each article individually if full_content_capture is enabled. The process_individual_article function calls browser_pool.render_and_screenshot to get the fully rendered page content and a screenshot, which are then used for deep content extraction and embedding.
The BrowserPool is instantiated and managed by the AppContext lifecycle, ensuring it is available to all processing tasks.
Sources: monitor/main.py:98-104, 255-285, monitor/feeds/base.py:410-436
Summary
The Fetcher Service is a critical, dual-capability system for content acquisition. It efficiently fetches structured feeds using a lightweight HTTP client and robustly renders complex article pages using a managed pool of headless browsers. This design ensures both performance for high-frequency checks and accuracy for deep content analysis, making it a foundational element of the monitoring pipeline.