/NickJLange/technical-blog-monitor

Wiki directory

Browse AI-generated documentation for any repository.

Troubleshooting

Relevant source files

The following files were used as context for generating this wiki page:

This guide provides solutions for common issues encountered during the setup, configuration, and operation of the Technical Blog Monitor. It covers problems ranging from dependency installation to runtime errors in the processing pipeline. For general testing procedures, refer to the Testing Guide.

The troubleshooting process generally involves checking configuration, examining logs for specific errors, and using command-line flags to isolate problems. The system is designed with structured logging and clear error messages to aid in diagnostics.

Sources: WARP.md, DEMO.md, TESTING.md

Common Issues and Solutions

This table summarizes frequent problems, their likely causes, and recommended solutions.

Symptom	Likely Cause	Fix	Source(s)
Playwright browser errors	Browsers are not installed or dependencies are missing.	Run `poetry run playwright install --force` to reinstall browsers and `poetry run playwright install-deps` to install system dependencies.	WARP.md:209-215
"No new posts found" on every run	The cache holds fingerprints of already processed posts.	Delete the cache directory (e.g., `rm -rf cache/`) to force reprocessing.	DEMO.md:104-105, WARP.md:219-220
`ImportError`	Python dependencies are missing or out of sync.	Run `uv sync` or `poetry install` to install the required packages.	DEMO.md:106-107, pyproject.toml
Headless browser blocked by `robots.txt` or bot detection	The target site is blocking automated requests.	Set `BROWSER__STEALTH_MODE=false` in your `.env` file and potentially adjust the user agent.	DEMO.md:108-109, monitor/config.py:53
Feed parsing errors	The feed URL is incorrect, or the feed's XML/HTML is invalid.	Verify the feed URL in your `.env` file. Run with `--log-level DEBUG` to see detailed parsing logs.	WARP.md:217-218, TESTING.md:120-122
Ollama connection errors	The Ollama service is not running locally.	Ensure the Ollama service is active by running `curl http://localhost:11434/api/tags` or starting it with `ollama serve`.	TESTING.md:108-115

Configuration Debugging

Configuration issues are a common source of errors. The system uses Pydantic for settings management, which provides detailed validation errors upon startup.

Sources: monitor/config.py, WARP.md:224-228

Validating Configuration

You can check your configuration without running the full application by executing a simple Python script. This will load settings from your .env file and immediately show any validation errors.

# Check configuration without running
poetry run python -c "from monitor.config import load_settings; print(load_settings())"

Sources: WARP.md:91-92

The following diagram illustrates the configuration loading and validation flow.

Generating architecture diagram...

This diagram shows how Pydantic validates environment variables against the Settings model, raising an error if the configuration is invalid.
Sources: monitor/config.py:292-295, monitor/config.py:121-125

Key Configuration Sections

Configuration is managed via environment variables, typically in a .env file, with nested structures delimited by a double underscore (__).

Prefix	Purpose	Relevant File
`FEEDS__`	Defines one or more blog feeds to monitor (name, URL, interval).	monitor/config.py:32-38
`BROWSER__`	Configures the Playwright headless browser (type, timeout, concurrency).	monitor/config.py:41-58
`CACHE__`	Sets up the caching layer (Redis URL, TTL, local path).	monitor/config.py:61-68
`EMBEDDING__`	Selects the embedding model (OpenAI, HuggingFace, etc.) and API keys.	monitor/config.py:81-106
`VECTOR_DB__`	Configures the vector database client (Qdrant, Chroma, etc.) and connection details.	monitor/config.py:128-145
`METRICS__`	Manages logging and Prometheus settings.	monitor/config.py:161-167

Sources: README.md:91-100

Runtime and Processing Issues

Errors can occur during the daemon's execution, such as when fetching feeds, rendering pages, or generating embeddings. The application uses structured logging to provide context for these errors.

Using Debugging Flags

The most effective way to diagnose runtime issues is to use the command-line arguments to run the monitor in a controlled manner.

Argument	Description	Source
`--once`	Runs the monitoring process once for all enabled feeds and then exits. Ideal for testing changes.	monitor/main.py:511-515
`--feed <name>`	Processes only a single, specified feed. Useful for isolating issues with a particular blog.	monitor/main.py:517-521
`--log-level <level>`	Sets the logging verbosity. Use `DEBUG` for detailed operational output.	monitor/main.py:523-527

Example: To debug a specific feed with maximum log output:

uv run monitor --feed "AWS Blog" --once --log-level DEBUG

Sources: TESTING.md:65-67

Error Handling Flow

The application includes retry logic for transient network errors and graceful shutdown procedures. If an error occurs while processing a post, it is logged, but it does not stop the entire feed processing task.

The following sequence diagram shows how an error during post processing is handled.

Generating architecture diagram...

This diagram shows that when process_post fails, the exception is caught and logged by process_feed, allowing the application to continue processing other posts in the queue.
Sources: monitor/main.py:326-332, monitor/main.py:372-376

Web Dashboard and Health Check

The application includes a web dashboard that can be used to monitor its status. If the dashboard is enabled, you can access it to see recent posts and basic statistics.

Dashboard URL: http://localhost:8080
Health Check Endpoint: http://localhost:8080/health

The health check endpoint provides a quick way to verify if the application is running and if it has successfully connected to the vector database.

{
  "status": "healthy",
  "timestamp": "2023-10-27T10:00:00Z",
  "vector_db_connected": true
}

A vector_db_connected: false status indicates a problem with the vector database configuration or connection.
Sources: monitor/web/app.py:175-181, monitor/dashboard.py:28-34

Testing and Quality Assurance

If you suspect a bug in the core logic, running the test suite can help identify the issue. The project includes unit tests, integration tests, and static analysis tools.

A single script can run all checks:

./run_all_tests.sh

This script executes pytest, ruff (linting), and mypy (type checking), providing a comprehensive overview of code health.
Sources: run_all_tests.sh:4-46, TESTING.md:24-33

Troubleshooting

Relevant source files

The following files were used as context for generating this wiki page:

Sources: WARP.md, DEMO.md, TESTING.md

Common Issues and Solutions

This table summarizes frequent problems, their likely causes, and recommended solutions.

Symptom	Likely Cause	Fix	Source(s)
Playwright browser errors	Browsers are not installed or dependencies are missing.	Run `poetry run playwright install --force` to reinstall browsers and `poetry run playwright install-deps` to install system dependencies.	WARP.md:209-215
"No new posts found" on every run	The cache holds fingerprints of already processed posts.	Delete the cache directory (e.g., `rm -rf cache/`) to force reprocessing.	DEMO.md:104-105, WARP.md:219-220
`ImportError`	Python dependencies are missing or out of sync.	Run `uv sync` or `poetry install` to install the required packages.	DEMO.md:106-107, pyproject.toml
Headless browser blocked by `robots.txt` or bot detection	The target site is blocking automated requests.	Set `BROWSER__STEALTH_MODE=false` in your `.env` file and potentially adjust the user agent.	DEMO.md:108-109, monitor/config.py:53
Feed parsing errors	The feed URL is incorrect, or the feed's XML/HTML is invalid.	Verify the feed URL in your `.env` file. Run with `--log-level DEBUG` to see detailed parsing logs.	WARP.md:217-218, TESTING.md:120-122
Ollama connection errors	The Ollama service is not running locally.	Ensure the Ollama service is active by running `curl http://localhost:11434/api/tags` or starting it with `ollama serve`.	TESTING.md:108-115

Configuration Debugging

Configuration issues are a common source of errors. The system uses Pydantic for settings management, which provides detailed validation errors upon startup.

Sources: monitor/config.py, WARP.md:224-228

Validating Configuration

You can check your configuration without running the full application by executing a simple Python script. This will load settings from your .env file and immediately show any validation errors.

# Check configuration without running
poetry run python -c "from monitor.config import load_settings; print(load_settings())"

Sources: WARP.md:91-92

The following diagram illustrates the configuration loading and validation flow.

Generating architecture diagram...

Key Configuration Sections

Configuration is managed via environment variables, typically in a .env file, with nested structures delimited by a double underscore (__).

Prefix	Purpose	Relevant File
`FEEDS__`	Defines one or more blog feeds to monitor (name, URL, interval).	monitor/config.py:32-38
`BROWSER__`	Configures the Playwright headless browser (type, timeout, concurrency).	monitor/config.py:41-58
`CACHE__`	Sets up the caching layer (Redis URL, TTL, local path).	monitor/config.py:61-68
`EMBEDDING__`	Selects the embedding model (OpenAI, HuggingFace, etc.) and API keys.	monitor/config.py:81-106
`VECTOR_DB__`	Configures the vector database client (Qdrant, Chroma, etc.) and connection details.	monitor/config.py:128-145
`METRICS__`	Manages logging and Prometheus settings.	monitor/config.py:161-167

Sources: README.md:91-100

Runtime and Processing Issues

Errors can occur during the daemon's execution, such as when fetching feeds, rendering pages, or generating embeddings. The application uses structured logging to provide context for these errors.

Using Debugging Flags

The most effective way to diagnose runtime issues is to use the command-line arguments to run the monitor in a controlled manner.

Argument	Description	Source
`--once`	Runs the monitoring process once for all enabled feeds and then exits. Ideal for testing changes.	monitor/main.py:511-515
`--feed <name>`	Processes only a single, specified feed. Useful for isolating issues with a particular blog.	monitor/main.py:517-521
`--log-level <level>`	Sets the logging verbosity. Use `DEBUG` for detailed operational output.	monitor/main.py:523-527

Example: To debug a specific feed with maximum log output:

uv run monitor --feed "AWS Blog" --once --log-level DEBUG

Sources: TESTING.md:65-67

Error Handling Flow

The following sequence diagram shows how an error during post processing is handled.

Generating architecture diagram...

Web Dashboard and Health Check

The application includes a web dashboard that can be used to monitor its status. If the dashboard is enabled, you can access it to see recent posts and basic statistics.

Dashboard URL: http://localhost:8080
Health Check Endpoint: http://localhost:8080/health

The health check endpoint provides a quick way to verify if the application is running and if it has successfully connected to the vector database.

{
  "status": "healthy",
  "timestamp": "2023-10-27T10:00:00Z",
  "vector_db_connected": true
}

A vector_db_connected: false status indicates a problem with the vector database configuration or connection.
Sources: monitor/web/app.py:175-181, monitor/dashboard.py:28-34

Testing and Quality Assurance

If you suspect a bug in the core logic, running the test suite can help identify the issue. The project includes unit tests, integration tests, and static analysis tools.

A single script can run all checks:

./run_all_tests.sh

This script executes pytest, ruff (linting), and mypy (type checking), providing a comprehensive overview of code health.
Sources: run_all_tests.sh:4-46, TESTING.md:24-33