Development Environment Setup

Relevant source files

The following files were used as context for generating this wiki page:

• pyproject.toml
• README.md
• WARP.md
• TESTING.md
• DEMO.md
• README.ja.md
• run_dashboard.sh
• monitor/config.py
• monitor/main.py

This guide provides comprehensive instructions for setting up a local development environment for the Technical Blog Monitor project. The project is a high-performance Python daemon designed to monitor technical blogs, render posts, extract content, and store embeddings in a vector database for semantic search.

Following these steps will ensure your environment is correctly configured with all necessary dependencies, tools, and configurations to run, test, and contribute to the project. For more detailed testing procedures, refer to the Testing Guide.

Sources: README.md:3-6, WARP.md:5-7

1. Prerequisites

Before starting, ensure your system meets the following requirements:

Requirement	Version/Tool	Notes	Source
Python	>=3.12	The project requires Python 3.12 or newer.	[pyproject.toml:6](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/pyproject.toml#L6)
Package Manager	uv or poetry	uv is recommended for speed. poetry is also supported.	[README.md:36](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L36), [WARP.md:11](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L11)
Git	N/A	Required for cloning the repository.	[README.md:32](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L32)

2. Installation

The installation process involves cloning the repository, installing dependencies, and setting up required tools like Playwright and pre-commit hooks.

This flowchart shows the general installation process.

Step-by-Step Instructions

1. Clone the repository:

   git clone https://github.com/NickJLange/technical-blog-monitor.git
   cd technical-blog-monitor
   

   Sources: [README.md:32-33](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L32-L33)

2. Install Dependencies:
   Choose one of the following package managers. uv is the preferred method mentioned in the main README.md.

   Using uv (Recommended):

   # Install uv if you don't have it
   curl -LsSf https://astral.sh/uv/install.sh | sh
   
   # Install main dependencies
   uv sync
   

   Sources: [README.md:35-39](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L35-L39)

   Using poetry:

   # Install main dependencies
   poetry install
   

   Sources: [WARP.md:11](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L11), [README.ja.md:9](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.ja.md#L9)

3. Install Playwright Browsers:
   This step is required to download the headless browser binaries used for rendering pages.

   # Using uv
   uv run playwright install
   
   # Using poetry
   poetry run playwright install
   

   Sources: [README.md:41-42](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L41-L42), [WARP.md:14](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L14)

4. Set Up Pre-commit Hooks:
   The project uses pre-commit hooks to automatically run linters and formatters (black, ruff, isort, mypy) before each commit.

   # Using uv
   uv sync --group dev
   uv run pre-commit install
   
   # Using poetry
   poetry install --with dev
   pre-commit install
   

   Sources: [README.md:102-104](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L102-L104), [WARP.md:16-17](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L16-L17)

3. Configuration

The application is configured using environment variables, which can be managed in a .env file.

1. Create a configuration file:
   Copy the example file to create your local configuration.

   cp .env.example .env
   

   Sources: [README.md:47](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L47)

2. Edit .env:
   Modify the .env file with your specific settings. Key sections include:

   • FEEDS__*__URL: The URLs of the blogs to monitor.
   • EMBEDDING__...: API keys for services like OpenAI or HuggingFace.
   • VECTOR_DB__...: Connection details for your vector database (e.g., Qdrant, Chroma).

   Sources: [README.md:51-54](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L51-L54), [WARP.md:101-106](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L101-L106)

Configuration Structure

Configuration is managed by Pydantic and loaded from the environment. Nested settings use a double underscore (__) as a delimiter.

Prefix	Purpose
FEEDS__	Definitions for multiple feeds (name, url, interval).
BROWSER__	Playwright options (headless, concurrency, stealth mode).
CACHE__	Caching layer settings (Redis URL, TTL).
EMBEDDING__	Embedding model selection, API keys, and batch size.
VECTOR_DB__	Vector database type, connection string, and collection name.
SCHEDULER__	Job scheduler options.
METRICS__	Prometheus and logging settings.

Sources: [README.md:88-96](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L88-L96), [monitor/config.py:270-280](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/monitor/config.py#L270-L280)

For example, to configure two feeds, you would use the FEEDS__N__ pattern:

# Feed 0
FEEDS__0__NAME="Google Cloud Blog"
FEEDS__0__URL="https://cloud.google.com/blog/products/rss"
FEEDS__0__CHECK_INTERVAL_MINUTES=60

# Feed 1
FEEDS__1__NAME="AWS Blog"
FEEDS__1__URL="https://aws.amazon.com/blogs/aws/feed/"
FEEDS__1__CHECK_INTERVAL_MINUTES=60

Sources: [WARP.md:101-104](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L101-L104), [TESTING.md:100-102](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/TESTING.md#L100-L102)

4. Running the Application

Once installed and configured, you can run the monitor in several modes.

Monitor Daemon

Command	Mode	Description
uv run monitor --once --log-level DEBUG	One-shot Debug	Processes all configured feeds once with detailed logging and then exits. Good for testing.
uv run monitor	Daemon	Runs indefinitely, polling feeds at their configured intervals. This is the production mode.
uv run monitor --feed "AWS Blog" --once	Specific Feed	Processes only the specified feed once. Useful for debugging a single source.

Sources: [README.md:59-69](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L59-L69), [WARP.md:22-25](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L22-L25)

Web Dashboard

The project includes a web dashboard to view monitored posts.

To run the dashboard:

# Using the provided script
./run_dashboard.sh

# Or directly with Python
export PYTHONPATH=.
python monitor/dashboard.py

The dashboard will be available at http://localhost:8080.

Sources: [run_dashboard.sh](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/run_dashboard.sh), [monitor/dashboard.py:28-36](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/monitor/dashboard.py#L28-L36)

Docker

The application is container-ready.

# Build the Docker image
docker build -t blog-monitor .

# Run the container with your environment file
docker run --env-file .env blog-monitor

Sources: [README.md:74-75](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L74-L75), [WARP.md:27-28](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L27-L28)

5. Testing and Quality Assurance

The development environment is equipped with a suite of tools for testing and code quality. These are typically run automatically by the pre-commit hooks, but can also be run manually.

Development Workflow

This diagram illustrates the typical developer workflow with automated quality checks.

Manual Checks

You can run these checks manually at any time.

Tool	Command (uv run ...)	Purpose
Pytest	pytest -q	Runs the entire test suite.
Ruff	ruff check .	Lints code for style and static analysis.
Mypy	mypy monitor/	Performs static type checking.
Black	black .	Formats Python code.
isort	isort .	Sorts import statements.
Bandit	bandit -r monitor -ll	Scans for common security vulnerabilities.

Sources: [WARP.md:30-40](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L30-L40), [README.md:107-112](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/README.md#L107-L112)

For more detailed testing instructions, including how to run tests with coverage, see TESTING.md.

6. Troubleshooting

Symptom	Likely Cause	Fix	Source
Playwright errors on run	Browser binaries are not installed or are corrupted.	Re-install the browsers: uv run playwright install --force	[WARP.md:143-144](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L143-L144), [DEMO.md:102](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/DEMO.md#L102)
"No new posts found" on every run	The cache is preventing re-processing of already seen posts.	Delete the local cache directory: rm -rf cache/	[WARP.md:149-150](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L149-L150), [DEMO.md:103](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/DEMO.md#L103)
ImportError	Dependencies are missing or out of sync.	Re-run the installation command: uv sync	[DEMO.md:104](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/DEMO.md#L104)
Configuration validation errors	The .env file has incorrect values or is missing required keys.	Check the .env file against .env.example and monitor/config.py.	[WARP.md:155-158](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L155-L158)

To debug configuration issues, you can run the following Python snippet to load and print the settings, which will show any Pydantic validation errors.

poetry run python -c "from monitor.config import load_settings; print(load_settings())"

Sources: [WARP.md:43-44](https://github.com/NickJLange/technical-blog-monitor/blob/HEAD/WARP.md#L43-L44)