Welcome to the Technical Blog Monitor

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• DEMO.md
• WARP.md
• monitor/main.py
• monitor/config.py
• monitor/web/app.py
• monitor/models/article.py
• test_basic.py
• monitor/web/templates/index.html
• TESTING.md

The Technical Blog Monitor is a high-performance Python daemon designed to track technical blogs from various companies. Its core function is to poll RSS/Atom feeds and JSON endpoints, render new posts in a headless browser, extract rich text and screenshots, and generate multimodal (text and image) embeddings. These embeddings are then stored in a pluggable vector database to enable powerful semantic search capabilities.

The system is built with a production-ready, modular architecture, leveraging asyncio and thread pools for high throughput. It is designed to be extensible, allowing developers to add new feed sources, content extractors, or vector database back-ends with minimal code changes. All aspects of the daemon are configurable through environment variables, managed by Pydantic for type safety and validation.

Sources: README.md:3-7, WARP.md:5-8

Key Features

The monitor provides a comprehensive set of features for building a multimodal ingestion pipeline for technical content.

Capability	Details
Hour-level monitoring	A configurable scheduler polls RSS/Atom feeds or JSON endpoints at user-defined intervals.
Async & Multithreaded	Built on asyncio and thread pools for optimal I/O and CPU throughput.
Headless rendering	Uses Playwright (Chromium, Firefox, or WebKit) to fully render pages and capture full-page screenshots.
Content extraction	A robust article parser based on readability-lxml turns raw HTML into clean text, metadata, and media links.
Caching layer	Pluggable Redis and local filesystem cache to avoid redundant downloads and re-processing.
Pluggable embeddings	Generate text and image embeddings with OpenAI, HuggingFace, Sentence-Transformers, or custom models.
Vector DB abstraction	Works with Qdrant, Chroma, Pinecone, Milvus, and Weaviate, chosen at runtime.
Observability	Features structured JSON logging, optional Prometheus metrics, and graceful shutdown.
Container-ready	Includes a multi-stage Dockerfile with Playwright browsers pre-installed.
Extensible	The modular codebase allows for easy addition of new feeds, extractors, or DB back-ends.

Sources: README.md:9-21

Architecture Overview

The system is designed with a modular, layered architecture that separates concerns into distinct components. The core processing pipeline discovers new posts, renders them, extracts content, generates embeddings, and finally stores the data.

This diagram illustrates the high-level data flow through the system's main components.

Core Components

• Application Context (monitor/main.py): The central orchestrator that manages the lifecycle of all components, including initialization and graceful shutdown. It holds shared resources like the thread pool, client instances, and the shutdown event.
  Sources: monitor/main.py:23-40
• Scheduler (APScheduler): Triggers feed processing jobs at configurable intervals. It uses a MemoryJobStore by default but can be configured to use Redis.
  Sources: monitor/main.py:130-150
• Feed Processor: Discovers new posts from RSS, Atom, or JSON feeds. It uses a caching layer to avoid reprocessing posts that have already been seen.
  Sources: WARP.md:44-48
• Browser Pool (Playwright): Manages a pool of headless browser instances for concurrently rendering web pages. It can capture full-page screenshots and is configured to handle timeouts and apply stealth techniques to avoid blocking.
  Sources: WARP.md:50-54, monitor/config.py:35-54
• Content Extractor: Uses readability-lxml and BeautifulSoup to parse rendered HTML, extracting clean article text, metadata (author, date, tags), and image URLs. This CPU-bound task is run in a ThreadPoolExecutor.
  Sources: monitor/extractor/article_parser.py:6-15, WARP.md:56-60
• Embedding Client: A pluggable component that generates vector embeddings for text and images. It supports providers like OpenAI and HuggingFace and includes batching and retry logic.
  Sources: WARP.md:62-65, monitor/config.py:90-120
• Vector DB Client: An abstract interface for interacting with various vector databases like Qdrant, Chroma, and others. It handles the upserting of embedding records.
  Sources: WARP.md:67-71, monitor/config.py:123-146

Processing Flow

The end-to-end processing for monitoring a feed is managed by the AppContext and executed by the scheduler. The following sequence diagram shows the interactions between components when a single post is processed.

This diagram shows the sequence of operations from job execution to data storage.
Sources: monitor/main.py:202-300

Core Data Models

The application relies on Pydantic models to structure data. The primary model is ArticleContent, which holds all information extracted from a blog post.

ArticleContent

This model represents the structured content extracted from an article's URL. It includes cleaned text, raw HTML, and rich metadata.

Other Models

• BlogPost: A lightweight model representing a post discovered from a feed, before full content extraction.
  Sources: test_basic.py:29
• EmbeddingRecord: Represents the final processed content, including text and image embeddings, ready for storage in the vector database.
  Sources: test_basic.py:118-132

Configuration

The application is configured via a .env file or environment variables, which are parsed into a typed Settings object using Pydantic. Nested configuration keys are delimited by a double underscore (__).

Sources: README.md:71-74, monitor/config.py:220-227

Main Configuration Sections

Prefix	Purpose	Source File
FEEDS__	Definitions for each blog feed (name, url, interval).	monitor/config.py:27
BROWSER__	Playwright options (headless, browser type, concurrency).	monitor/config.py:35
CACHE__	Redis URL, TTL, and local cache path.	monitor/config.py:57
EMBEDDING__	Model selection (OpenAI, etc.), API keys, and batch size.	monitor/config.py:90
VECTOR_DB__	DB type, connection string, and collection name.	monitor/config.py:123
SCHEDULER__	APScheduler store and timing options.	monitor/config.py:149
METRICS__	Prometheus and structured logging settings.	monitor/config.py:157
WEB_DASHBOARD__	Settings for the optional web UI.	monitor/config.py:166

Sources: README.md:78-86, monitor/config.py

Example Feed Configuration

The following demonstrates how to define a feed in the .env file. Multiple feeds can be added by incrementing the index (e.g., FEEDS__1__...).

# Feed Configuration (can have multiple feeds with incrementing numbers)
FEEDS__0__NAME="Google Cloud Blog"
FEEDS__0__URL="https://cloud.google.com/blog/products/rss"
FEEDS__0__CHECK_INTERVAL_MINUTES=60
FEEDS__0__ENABLED=true

Sources: WARP.md:100-103

Web Dashboard

The project includes an optional web dashboard built with FastAPI to view monitoring statistics and browse recently processed posts.

The dashboard is defined in monitor/web/app.py and can be launched via run_dashboard.sh. It provides a user-friendly interface rendered from monitor/web/templates/index.html. The UI displays key stats like total posts, posts today, and active sources, and lists recent articles with links to the original source. Data is fetched dynamically from the API endpoints.

Sources: monitor/web/app.py:5-8, monitor/dashboard.py:10-13, monitor/web/templates/index.html:8-12

API Endpoints

The dashboard is powered by a simple REST API.

Endpoint	Method	Description
/	GET	Renders the main HTML dashboard page.
/api/stats	GET	Returns dashboard statistics like post counts and active sources.
/api/posts	GET	Returns a paginated list of recently processed posts.
/health	GET	A health check endpoint that reports service status.

Sources: monitor/web/app.py:72-200

Getting Started

Installation and Setup

1. Clone the repository and install dependencies:

   git clone https://github.com/NickJLange/technical-blog-monitor.git
   cd technical-blog-monitor
   uv sync
   

2. Install Playwright browsers:

   uv run playwright install
   

3. Configure the monitor:
   Copy the example environment file and edit it with your API keys and database settings.

   cp .env.example .env
   

Sources: README.md:32-51

Running the Monitor

• Run once for debugging:
  This command processes all enabled feeds once and then exits.

  uv run monitor --once --log-level DEBUG
  

• Run in daemon mode:
  This command runs the monitor continuously as a background service.

  uv run monitor
  

Sources: README.md:55-61

Testing

The repository includes a comprehensive test suite.

• Run all unit tests with pytest:

  uv run pytest -q
  

• Run a basic integration test script:
  This script verifies the core components work together without heavy dependencies.

  uv run python test_basic.py
  

Sources: TESTING.md:18-32, README.md:105-107

Conclusion

The Technical Blog Monitor is an end-to-end ingestion pipeline for consuming, processing, and searching content from technical blogs. Its asynchronous, modular, and extensible design makes it a powerful tool for developers and organizations looking to stay updated with the latest engineering articles and build semantic search applications on top of this content. By leveraging modern Python libraries and a pluggable architecture, it offers a flexible and scalable solution for multimodal data processing.

Sources: DEMO.md:104-106, README.md:3-7