Feature: Feed Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
The Feed Monitoring feature is the primary data ingestion mechanism for the Technical Blog Monitor. Its core responsibility is to periodically check configured blog feeds, identify new posts, and parse them into a standardized BlogPost format for downstream processing. The system is designed with a pluggable architecture to support various feed formats like RSS, Atom, and JSON, and includes mechanisms for caching, change detection, and error handling to ensure efficient and reliable operation.
This feature is orchestrated by the main application scheduler, which triggers feed processing jobs at configurable intervals. Each job executes a pipeline that involves fetching feed data, parsing it according to its format, and filtering out posts that have already been processed.
Sources: monitor/main.py:171-184, monitor/feeds/base.py:1-13
Architecture and Core Components
The feed monitoring system is built around a FeedProcessor abstract base class, with concrete implementations for different feed types. A factory function, get_feed_processor, automatically selects the correct processor based on URL patterns or content analysis. The entire process is managed by the process_feed_posts function.
Sources: monitor/feeds/base.py:40-47, monitor/feeds/base.py:183-196
Feed Processing Flow
The following diagram illustrates the high-level workflow for processing a single feed.
This diagram shows the sequence from the scheduler's trigger to the final output of new BlogPost objects ready for the next stage of the pipeline.
Sources: monitor/main.py:228-243, monitor/feeds/base.py:252-328
FeedProcessor Class Hierarchy
The system uses an abstract base class FeedProcessor to define a common interface for handling different feed formats. This allows for easy extension to support new feed types in the future.
This diagram illustrates that RSSFeedProcessor, AtomFeedProcessor, and JSONFeedProcessor are concrete implementations of the FeedProcessor interface.
Sources: monitor/feeds/base.py:40-101, monitor/feeds/rss.py:27, monitor/feeds/atom.py:27, monitor/feeds/json.py:27
Key Functions
| Function | Description | Source File |
|---|---|---|
process_feed_posts() | Orchestrates the entire feed processing pipeline, from getting a processor to discovering new posts. | monitor/feeds/base.py |
get_feed_processor() | Factory function that inspects a feed's URL and content to return the appropriate FeedProcessor instance (RSS, Atom, or JSON). | monitor/feeds/base.py |
discover_new_posts() | Fetches feed content, generates a fingerprint to detect changes, parses the feed, and filters out posts that have already been seen using a cache. | monitor/feeds/base.py |
parse_feed_entries() | A common helper function that converts raw entries from any feed type into a standardized list of BlogPost objects. | monitor/feeds/base.py |
Sources: monitor/feeds/base.py:183-250, monitor/feeds/base.py:252-328, monitor/feeds/base.py:331-447, monitor/feeds/base.py:450-544
Data Flow Sequence
The interaction between different components during a feed check is crucial. The scheduler in main.py initiates the process, which then involves the feed processor, cache client, and HTTP client to identify new posts.
This sequence shows how discover_new_posts uses a fingerprint and post-level caching to efficiently determine what content is new.
Sources: monitor/main.py:228-243, monitor/feeds/base.py:252-328
Configuration
Feed monitoring is configured via the FeedConfig model. Multiple feeds can be defined using environment variables with a FEEDS__N__ prefix, where N is an index.
Sources: monitor/config.py:30-37, monitor/config.py:302-311
FeedConfig Model
The FeedConfig Pydantic model defines the settings for a single feed source.
| Field | Type | Description | Default |
|---|---|---|---|
name | str | A unique name for the feed. | - |
url | AnyHttpUrl | The URL of the feed (RSS, Atom, JSON, or HTML page). | - |
check_interval_minutes | int | How often the scheduler should check this feed. | 60 |
max_posts_per_check | int | The maximum number of posts to process in a single run. | 10 |
headers | Dict[str, str] | Custom HTTP headers to use when fetching the feed. | {} |
enabled | bool | A flag to enable or disable monitoring for this feed. | True |
Sources: monitor/config.py:30-37
Example Configuration (.env)
# Feed 0: Google Cloud Blog (RSS)
FEEDS__0__NAME="Google Cloud Blog"
FEEDS__0__URL="https://cloud.google.com/blog/products/rss"
FEEDS__0__CHECK_INTERVAL_MINUTES=60
# Feed 1: Uber Engineering Blog (HTML page with auto-discovery)
FEEDS__1__NAME="Uber Engineering Blog"
FEEDS__1__URL="https://www.uber.com/en-US/blog/engineering/"
FEEDS__1__ENABLED=true
This example demonstrates how to configure multiple feeds using the nested environment variable syntax. The system automatically handles discovering the actual feed link from an HTML page, as shown for the Uber blog.
Sources: monitor/config.py:289-296, test_feeds.py:39-54
Data Models
BlogPost
The primary output of the feed monitoring process is the BlogPost object. This model standardizes the information extracted from various feed formats.
| Field | Type | Description |
|---|---|---|
id | str | A unique SHA256 hash generated from the feed name and entry ID. |
url | HttpUrl | The canonical URL of the blog post. |
title | str | The title of the post. |
source | str | The name of the feed this post came from. |
author | Optional[str] | The author of the post. |
publish_date | Optional[datetime] | The original publication date. |
updated_date | Optional[datetime] | The date the post was last updated. |
summary | Optional[str] | A short summary or description of the post. |
tags | List[str] | A list of tags or categories associated with the post. |
metadata | Dict[str, Any] | A dictionary for storing additional, non-standard data. |
Sources: monitor/models/blog_post.py:16-47
Conclusion
The Feed Monitoring feature is a robust and extensible system for discovering new content from a variety of sources. Its use of a pluggable FeedProcessor architecture, automatic feed type detection, and efficient change detection via caching makes it a foundational component of the application's data ingestion pipeline. By normalizing disparate feed formats into a consistent BlogPost model, it provides a clean and reliable stream of data for subsequent processing stages like content extraction and embedding generation.
Sources: monitor/feeds/base.py, monitor/main.py