Data Models

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/models/article.py
• monitor/models/blog_post.py
• monitor/models/embedding.py
• monitor/models/cache_entry.py
• monitor/models/content_type.py
• monitor/models/init.py
• monitor/web/app.py
• monitor/extractor/article_parser.py
• WARP.md

The data models are the structural foundation of the technical-blog-monitor application. They define the shape and behavior of the core entities that flow through the processing pipeline, from initial discovery to final storage. Built using Pydantic, these models ensure data integrity, provide validation, and offer a clear, type-hinted representation of data at each stage.

The primary models represent a blog post as it is discovered (BlogPost), its fully extracted and cleaned content (ArticleContent), and its final vector representation ready for semantic search (EmbeddingRecord). These models are central to the system's modular architecture, enabling clear separation of concerns between feed processing, content extraction, embedding generation, and vector storage.

Core Entities and Relationships

The system revolves around three main data models: BlogPost, ArticleContent, and EmbeddingRecord. They represent the progression of data through the ingestion pipeline.

The following diagram illustrates the relationships between these core entities.

This diagram shows that a BlogPost is discovered first. Its URL is then used to create an ArticleContent object. Finally, the ArticleContent is processed to generate an EmbeddingRecord for storage.

Sources: monitor/models/blog_post.py, monitor/models/article.py, monitor/models/embedding.py

Data Transformation Flow

The data models are used sequentially in the processing pipeline. The following flowchart shows how an initial feed entry is transformed into a searchable vector record.

Detailed Model Definitions

BlogPost

Represents a blog post as discovered from an RSS, Atom, or JSON feed. It contains metadata from the feed and internal state for tracking its processing status.

Sources: monitor/models/blog_post.py

Field	Type	Description
id	str	A unique SHA256 hash generated from the URL and title.
url	HttpUrl	The canonical URL of the blog post.
title	str	The title of the post.
source	str	The name of the feed this post was discovered from.
author	Optional[str]	The author of the post, if available.
publish_date	Optional[datetime]	The original publication date.
updated_date	Optional[datetime]	The last modification date.
summary	Optional[str]	A short summary from the feed.
content_type	ContentType	The classified type of content (e.g., BLOG_POST).
tags	List[str]	A list of tags or categories.
discovered_at	datetime	Timestamp of when the post was first discovered.
processing_status	str	Internal status, e.g., "pending", "processed".
fetch_attempts	int	Counter for fetch attempts.

Key Methods:

• generate_id(): Creates a unique ID based on the post's URL and title.
• normalize_tags(): A class method validator that cleans and de-duplicates tags.
• ensure_timezone(): A class method validator that ensures all datetime objects are timezone-aware (UTC).

Sources: monitor/models/blog_post.py:22-81

ArticleContent

Represents the fully processed content extracted from a blog post's URL. This model holds the cleaned text, raw HTML, and rich metadata obtained after rendering the page in a headless browser.

Sources: monitor/models/article.py

Field	Type	Description
url	HttpUrl	The URL of the article.
title	str	The extracted title of the article.
text	str	The cleaned, plain text content of the article.
html	str	The raw HTML of the main article body.
author	Optional[str]	The extracted author name.
publish_date	Optional[datetime]	The extracted publication date.
word_count	int	The total number of words in the text.
image_urls	List[str]	A list of absolute URLs for images found in the content.
screenshot_paths	List[str]	Local filesystem paths to screenshots of the rendered page.
tags	List[str]	A list of extracted tags.
reading_time_minutes	Optional[float]	Estimated reading time, calculated automatically from word_count.
extracted_at	datetime	Timestamp of when the content was extracted.
language	Optional[str]	Detected language of the article text.

Key Methods:

• calculate_reading_time(): A model validator that computes the reading time based on an average of 225 words per minute.
• _sanitize_text(): A field validator that removes excessive whitespace and newlines from the text content.
• generate_id(): Generates a unique ID from the URL and title.
• resolve_relative_urls(): Converts relative image URLs to absolute URLs.

Sources: monitor/models/article.py:21-127

EmbeddingRecord

This is the final data structure optimized for storage and retrieval in a vector database. It contains the essential metadata and the generated text and image embeddings.

Sources: monitor/models/embedding.py

Field	Type	Description
id	str	A unique identifier for the record, typically matching the BlogPost ID.
url	HttpUrl	The URL of the original article.
title	str	The title of the article.
publish_date	Optional[datetime]	The publication date.
text_embedding	Optional[List[float]]	The vector embedding of the article's text content.
image_embedding	Optional[List[float]]	The vector embedding of the article's screenshot or main image.
metadata	Dict[str, Any]	A dictionary for storing additional, non-indexed metadata like author, tags, and word count.

This model is referenced in monitor/main.py when a post is successfully processed and ready to be stored.
Sources: monitor/main.py:291-310

CacheEntry

A generic model used by the caching layer to wrap cached items with metadata.

Sources: monitor/models/cache_entry.py

Field	Type	Description
key	str	The unique cache key.
value	Any	The data being cached.
created_at	datetime	Timestamp when the cache entry was created.
expires_at	Optional[datetime]	Timestamp when the cache entry will expire.

Enumerations

ContentType

This enumeration is used to classify the type of content discovered. The classification is performed in monitor/extractor/article_parser.py based on keywords in the title and URL.

Sources: monitor/models/content_type.py, monitor/extractor/article_parser.py:588-620

Value	Description
ARTICLE	A standard article.
BLOG_POST	A post from a blog.
TUTORIAL	A tutorial or how-to guide.
DOCUMENTATION	Technical documentation or reference material.
RELEASE_NOTES	Release notes, changelogs, or version updates.
CASE_STUDY	A case study or success story.
NEWS	A news item or announcement.
VIDEO	A video post.
PODCAST	A podcast episode.
UNKNOWN	The content type could not be determined.

Web Dashboard Models

The web dashboard (monitor/web/app.py) uses its own set of Pydantic models to serialize data for the frontend API.

Sources: monitor/web/app.py

PostSummary

A lightweight model for displaying a list of recent posts on the dashboard. It aggregates data from the EmbeddingRecord.

Sources: monitor/web/app.py:21-32

Field	Type	Description
id	str	The post's unique ID.
title	str	The post's title.
url	str	The post's URL.
source	str	The name of the blog or source.
author	Optional[str]	The author's name.
publish_date	Optional[datetime]	The publication date.
summary	Optional[str]	A short summary of the post.
tags	List[str]	Associated tags.
word_count	Optional[int]	The word count of the article.

DashboardStats

A model to provide aggregate statistics for the main dashboard view.

Sources: monitor/web/app.py:35-42

Field	Type	Description
total_posts	int	Total number of posts processed.
posts_today	int	Number of posts processed today.
posts_week	int	Number of posts processed this week.
sources	List[str]	A list of unique feed sources.
latest_update	Optional[datetime]	Timestamp of the last update.

Conclusion

The data models in monitor/models/ are a critical component of the application, providing structure, validation, and clarity to the data as it moves through the ingestion pipeline. By using Pydantic, the system ensures that data at every step is well-formed and consistent. The separation into BlogPost, ArticleContent, and EmbeddingRecord clearly delineates the different stages of data processing, contributing to a modular and maintainable codebase. These models are re-exported from the monitor.models package for stable and consistent import paths across the project.

Sources: monitor/models/init.py