Adding a New Feed Source

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/config.py
• monitor/feeds/base.py
• monitor/feeds/rss.py
• monitor/feeds/atom.py
• monitor/main.py
• test_feeds.py
• monitor/models/blog_post.py

This document outlines the process for adding a new blog feed source to the Technical Blog Monitor. The system is designed to be extensible, primarily through configuration, but also allows for custom feed processing logic if required. A "feed source" is any URL that provides a list of blog posts, typically in RSS, Atom, or JSON format.

The core of the feed processing system is built around a pluggable architecture that automatically detects the feed type and uses the appropriate parser. For most standard feeds, adding a new source is a matter of adding a few lines to the application's configuration. For non-standard sources, a custom FeedProcessor can be implemented.

Configuration-Based Approach

The primary method for adding a new feed is by defining it in the configuration, which is loaded from environment variables or a .env file. The system uses a Pydantic model, FeedConfig, to validate and structure the feed settings.

Sources: monitor/config.py:28-35

FeedConfig Model

Each feed is defined by a FeedConfig object. The fields for this model are detailed below.

Field	Type	Default	Description
name	str	-	A unique, human-readable name for the feed.
url	AnyHttpUrl	-	The URL of the feed (RSS, Atom, JSON, or HTML page with a feed link).
check_interval_minutes	int	60	How often the scheduler should check this feed for new posts.
max_posts_per_check	int	10	The maximum number of recent posts to process in a single check.
headers	Dict[str, str]	{}	Custom HTTP headers to send when fetching the feed.
enabled	bool	True	A flag to enable or disable processing for this feed.

Sources: monitor/config.py:28-35

Environment Variable Configuration

Feeds are configured as a list, using a double-underscore __ as a nested delimiter in environment variables. You can define multiple feeds by incrementing the index (e.g., FEEDS__0__, FEEDS__1__).

# Feed 0: Google Cloud Blog
FEEDS__0__NAME="Google Cloud Blog"
FEEDS__0__URL="https://cloud.google.com/blog/products/rss"
FEEDS__0__CHECK_INTERVAL_MINUTES=60

# Feed 1: Uber Engineering Blog
FEEDS__1__NAME="Uber Engineering Blog"
FEEDS__1__URL="https://www.uber.com/en-US/blog/engineering/"
FEEDS__1__ENABLED=true

This structure is parsed into a list of FeedConfig objects within the main Settings model.

Sources: monitor/config.py:302-311

Feed Processing Architecture

Once a feed is configured, the system uses a modular architecture to fetch, parse, and extract posts. This is orchestrated by the process_feed function in monitor/main.py and implemented in the monitor/feeds/ package.

FeedProcessor Abstraction

The system uses an abstract base class, FeedProcessor, to define a common interface for handling different feed formats. The project includes concrete implementations for RSS, Atom, and JSON feeds.

This diagram shows the class hierarchy for feed processors.

Automatic Feed Type Detection

A key feature is the get_feed_processor function, which automatically determines the correct FeedProcessor to use for a given FeedConfig. This eliminates the need to specify the feed type in the configuration.

The detection logic follows these steps:

1. URL Pattern Matching: Checks if the URL contains common patterns like /rss, /atom.xml, or .json.
2. Content-Type Header: If the URL pattern is not decisive, it fetches the feed and inspects the Content-Type HTTP header (e.g., application/rss+xml).
3. Content Sniffing: As a final fallback, it inspects the raw content for XML tags like <rss> or <feed>, or checks if the content is valid JSON.
4. Default: If all else fails, it defaults to RSSFeedProcessor.

The following flowchart illustrates the auto-detection process.

Data Flow for New Post Discovery

When the scheduler triggers a job for a feed, a sequence of operations is executed to discover new posts while avoiding redundant processing.

This sequence diagram shows the interactions between components during a feed check.

Key steps in the flow:

1. The Scheduler calls process_feed for a configured feed.
2. discover_new_posts orchestrates the discovery.
3. The appropriate FeedProcessor fetches the feed content.
4. A fingerprint (a hash or latest entry ID) of the content is generated to detect changes.
5. If the fingerprint matches the one in the Cache, processing stops.
6. If the feed has changed, it's parsed, and posts are extracted into BlogPost objects.
7. Each post's ID is checked against the cache to filter out already-processed posts.
8. Only new, unseen posts are returned for further processing (content extraction, embedding, etc.).
9. The new feed fingerprint is stored in the cache.

Sources: monitor/main.py:207-227, monitor/feeds/base.py:270-368, monitor/models/blog_post.py

Implementing a Custom Feed Processor

If a target site does not provide a standard RSS, Atom, or JSON feed, you can create a custom processor by inheriting from FeedProcessor.

1. Create a new Python file in monitor/feeds/.

2. Define a class that inherits from monitor.feeds.base.FeedProcessor.

3. Implement the following abstract methods:

   • async def fetch_feed(self, client: httpx.AsyncClient) -> bytes:
     • Logic to fetch the raw data from the source. This might involve scraping an HTML page.
   • async def parse_feed(self, content: bytes) -> List[Dict[str, Any]]:
     • Logic to parse the raw content into a list of dictionaries, where each dictionary represents a single post.
   • async def extract_posts(self, entries: List[Dict[str, Any]]) -> List[BlogPost]:
     • Logic to convert the list of dictionaries from parse_feed into a list of BlogPost model objects. The parse_feed_entries helper function can be used here.

4. Update the get_feed_processor function in monitor/feeds/base.py to include logic that recognizes and instantiates your new custom processor.

Sources: monitor/feeds/base.py:40-92

Testing a New Feed

After adding a feed to your configuration, you can test it without running the full daemon.

• Unit Testing: The test_feeds.py script provides a template for testing feed parsing in isolation. It uses a DummyBrowserPool and MemoryCacheClient to focus solely on the feed processing logic. You can add your new feed's configuration to the BLOG_CONFIGS list in this file to test it.
  Sources: test_feeds.py:41-58, 80-120

• One-off Run: The application's main entry point supports command-line arguments for targeted runs. Use the --once and --feed flags to process only your new feed and then exit. This is useful for end-to-end testing and debugging.

  # Run once, processing only the "Uber Engineering Blog" feed with DEBUG logging
  poetry run monitor --once --feed "Uber Engineering Blog" --log-level DEBUG
  

  Sources: monitor/main.py:465-476, 497-503

Summary

Adding a new feed source is a straightforward process. For standard RSS/Atom/JSON feeds, it only requires adding environment variables. The system's architecture handles the detection and processing automatically. For non-standard feeds, the FeedProcessor abstract base class provides a clear and extensible pattern for implementing custom logic, ensuring that any web-based source of articles can be integrated into the monitoring pipeline.