Quick Start Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• pyproject.toml
• sites.txt
• monitor/main.py
• monitor/config.py
• WARP.md
• DEMO.md
• TESTING.md
• monitor/web/app.py
• run_dashboard.sh
• monitor/web/templates/index.html

This guide provides instructions for setting up, configuring, and running the Technical Blog Monitor. The project is a high-performance Python daemon designed to track technical blogs, render new posts in a headless browser, extract content, generate multimodal embeddings, and store them in a vector database for semantic search.

The system is built with an asynchronous architecture using asyncio and thread pools for high throughput. It features a pluggable design for components like embedding models and vector databases, making it extensible and adaptable to different environments. For more detailed testing information, see the Testing Guide.

Sources: README.md:3-7, WARP.md:5-7, DEMO.md:5-15

Installation and Setup

1. Clone the Repository

First, clone the project from GitHub.

git clone https://github.com/NickJLange/technical-blog-monitor.git
cd technical-blog-monitor

Sources: README.md:40-41

2. Install Dependencies

The project uses uv for dependency management. Install the required packages from pyproject.toml.

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dependencies
uv sync

Alternatively, if using poetry:

# Install dependencies with Poetry
poetry install

Sources: README.md:43-49, WARP.md:11-12, pyproject.toml:7-39

3. Install Playwright Browsers

The monitor uses Playwright for headless browser rendering. You must install the necessary browser binaries.

# Using uv
uv run playwright install

# Using poetry
poetry run playwright install

Sources: README.md:51-52, WARP.md:15-16

Configuration

The application is configured using environment variables, typically managed in a .env file. All settings are defined and validated using Pydantic models in monitor/config.py.

Sources: README.md:65-69, monitor/config.py

1. Create a .env file

Copy the example environment file to create your own configuration.

cp .env.example .env

Sources: README.md:56-57

2. Set Key Configuration Values

Edit the .env file to set up your environment. Nested configuration keys use a double underscore (__) as a delimiter.

Key Configuration Sections

Prefix	Purpose	Source File
FEEDS__	Defines the blog feeds to monitor (name, URL, interval).	monitor/config.py:30-36
BROWSER__	Configures Playwright options like headless mode and concurrency.	monitor/config.py:39-58
CACHE__	Sets up the caching layer (e.g., Redis URL, TTL).	monitor/config.py:61-70
EMBEDDING__	Selects the embedding model (e.g., OpenAI, HuggingFace) and API keys.	monitor/config.py:80-111
VECTOR_DB__	Configures the vector database type, connection, and collection name.	monitor/config.py:119-138
WEB_DASHBOARD__	Toggles and configures the optional web dashboard.	monitor/config.py:157-162

Sources: README.md:71-78, WARP.md:104-105

Example .env Configuration

# Feed Configuration (use FEEDS__0__, FEEDS__1__, etc. for multiple feeds)
FEEDS__0__NAME="Google Cloud Blog"
FEEDS__0__URL="https://cloud.google.com/blog/products/rss"
FEEDS__0__CHECK_INTERVAL_MINUTES=60

# Embedding Configuration
EMBEDDING__TEXT_MODEL_TYPE=openai
EMBEDDING__OPENAI_API_KEY=sk-your-key-here
EMBEDDING__TEXT_MODEL_NAME=text-embedding-ada-002

# Vector Database Configuration
VECTOR_DB__DB_TYPE=qdrant
VECTOR_DB__CONNECTION_STRING=http://localhost:6333
VECTOR_DB__COLLECTION_NAME=technical_blog_posts

Sources: WARP.md:108-121, monitor/config.py:228-235

A list of potential sites to monitor can be found in sites.txt.

Sources: sites.txt

Running the Application

The application can be run in several modes. The main script is defined in pyproject.toml and executed via monitor.main:main.

Sources: pyproject.toml:42-43, monitor/main.py:465-507

One-Shot Debug Run

Process all configured feeds once and then exit. This is useful for testing your configuration. The --once flag triggers this mode.

# Using uv
uv run monitor --once --log-level DEBUG

# Using poetry
poetry run monitor --once --log-level DEBUG

Sources: README.md:61-62, monitor/main.py:485-489

Process a Specific Feed

You can target a single feed by name using the --feed argument. The feed name must match a name defined in your FEEDS__ configuration.

# Using uv
uv run monitor --feed "Google Cloud Blog" --once

# Using poetry
poetry run monitor --feed "Google Cloud Blog" --once

Sources: README.md:80-81, monitor/main.py:491-495

Daemon Mode

Run the monitor as a continuous service. It will poll the feeds at their configured intervals.

# Using uv
uv run monitor

# Using poetry
poetry run monitor

Sources: README.md:64-65, WARP.md:25-26

Docker

The repository includes a Dockerfile for containerized deployment.

docker build -t blog-monitor .
docker run --env-file .env blog-monitor

Sources: README.md:67-70, WARP.md:28-30

High-Level Architecture

The application follows a modular processing pipeline, orchestrated by the main application context.

This diagram shows the flow of data from feed discovery to storage in the vector database.

Web Dashboard

The project includes an optional web dashboard built with FastAPI to view statistics and recently processed posts.

Running the Dashboard

You can start the dashboard using the provided script.

./run_dashboard.sh

This will start a web server, typically on http://localhost:8080.

Sources: run_dashboard.sh, monitor/dashboard.py:16-37

Dashboard Features

The dashboard provides:

• A homepage with summary statistics.
• API endpoints to fetch data programmatically.

This diagram shows the interaction between the user, the dashboard, and the backend services.

API Endpoints

Endpoint	Method	Description
/	GET	Serves the main HTML dashboard page.
/api/stats	GET	Returns DashboardStats including total posts, posts today/week, and active sources.
/api/posts	GET	Returns a paginated list of PostSummary objects for recent posts.
/health	GET	A health check endpoint that reports application status.

Sources: monitor/web/app.py:73-187

Development and Testing

For development, it's recommended to set up pre-commit hooks to ensure code quality.

Pre-commit Setup

# Install dev dependencies
uv sync --group dev

# Install hooks
uv run pre-commit install

Sources: README.md:91-94, WARP.md:18-19

Running Tests

The test suite uses pytest.

# Run all tests
uv run pytest -q

# Run tests with coverage
uv run pytest -q --cov=monitor --cov-report=term-missing

Sources: README.md:96-97, WARP.md:33-36, TESTING.md:26-27