Configuration

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/config.py
• monitor/main.py
• README.md
• test_basic.py
• WARP.md
• monitor/web/app.py

The technical-blog-monitor uses a robust, Pydantic-based configuration system to manage all application settings. This system provides strong type validation, default values, and flexible loading from environment variables or .env files. The central definition for all settings is located in monitor/config.py, which ensures a single source of truth for configuration across the application.

All settings are loaded into a main Settings object, which is then passed to the AppContext to be available throughout the application's lifecycle. The configuration is designed to be hierarchical, using a double underscore (__) as a delimiter for nested settings, making it easy to manage complex configurations via environment variables.

Sources: monitor/config.py, monitor/main.py:53-61, README.md:65-80

Core Concepts

The configuration system is built on pydantic-settings, which extends Pydantic's data validation capabilities to settings management.

Sources: monitor/config.py:20-25

Loading Mechanism

Settings are loaded by calling the load_settings() function, which instantiates the main Settings class. The Settings class is configured via SettingsConfigDict to:

1. Read variables from a .env file.
2. Use a double underscore (__) as a delimiter for nested environment variables.
3. Ignore any extra environment variables that are not defined in the models.

This process is initiated at application startup within monitor/main.py.

Sources: monitor/config.py:293-299, monitor/main.py:466

The following diagram illustrates the configuration loading flow.

Hierarchical Structure

Nested configuration models are defined as environment variables using a __ delimiter. For example, to set the headless option within the BrowserConfig, the environment variable would be BROWSER__HEADLESS=true.

A special case exists for lists, such as the feeds configuration. These are defined using an indexed pattern, like FEEDS__0__NAME="...", FEEDS__0__URL="...", FEEDS__1__NAME="...", etc. A custom validator, _coerce_feeds_from_mapping, handles the conversion of this dictionary-like structure into a List[FeedConfig].

Sources: WARP.md:112-124, monitor/config.py:302-311

Validation

Pydantic automatically validates that all settings conform to their defined types. The system also employs custom validators for more complex logic. For example:

• EmbeddingConfig validates that an API key is provided if the corresponding model type (e.g., openai) is selected.
• VectorDBConfig validates the format of the connection_string.
• Settings validates that all feed.name values are unique.

Sources: monitor/config.py:122-127, monitor/config.py:171-178, monitor/config.py:313-318

Main Configuration Classes

The configuration is structured into several Pydantic BaseModel classes, with Settings being the top-level container.

This diagram shows the relationship between the main Settings class and its nested configuration objects.

Component Configuration Details

Each major component of the application has its own dedicated configuration model.

FeedConfig

Defines the settings for a single feed source.

Attribute	Type	Default	Description
name	str	-	Unique name for the feed.
url	AnyHttpUrl	-	The URL of the RSS/Atom/JSON feed.
check_interval_minutes	int	60	How often to check the feed for new posts.
max_posts_per_check	int	10	Maximum number of posts to process in a single check.
headers	Dict[str, str]	{}	Custom HTTP headers for fetching the feed.
enabled	bool	True	Whether the feed is active.
Sources: [monitor/config.py:29-35]

BrowserConfig

Configuration for the headless Playwright browser used for rendering pages.

Attribute	Type	Default	Description
headless	bool	True	Run the browser in headless mode.
browser_type	str	"chromium"	Browser to use (chromium, firefox, webkit).
timeout_seconds	int	30	Page load timeout.
max_concurrent_browsers	int	3	Maximum number of browser instances in the pool.
block_ads	bool	True	Enable ad blocking.
stealth_mode	bool	True	Enable stealth mode to avoid bot detection.
screenshot_dir	Optional[Path]	None	Directory to store screenshots. Defaults to <cwd>/data/screenshots.
Sources: [monitor/config.py:38-60]

CacheConfig

Settings for the caching layer (Redis or local filesystem).

Attribute	Type	Default	Description
enabled	bool	True	Enable or disable caching.
redis_url	Optional[str]	None	Connection URL for Redis. If not set, filesystem cache is used.
cache_ttl_hours	int	168 (7 days)	Default time-to-live for cache entries.
local_storage_path	Path	./cache	Path for the local filesystem cache.
Sources: [monitor/config.py:75-81]

EmbeddingConfig

Configuration for generating text and image embeddings.

Attribute	Type	Default	Description
text_model_type	EmbeddingModelType	openai	The type of embedding model to use (openai, huggingface, etc.).
text_model_name	str	text-embedding-ada-002	The specific name of the text embedding model.
image_model_type	Optional[EmbeddingModelType]	None	The type of image embedding model.
openai_api_key	Optional[SecretStr]	None	API key for OpenAI services.
huggingface_api_key	Optional[SecretStr]	None	API key for HuggingFace services.
embedding_dimensions	Optional[int]	None	The dimension of the text embedding vectors.
Sources: [monitor/config.py:102-121]

VectorDBConfig

Settings for the vector database client.

Attribute	Type	Default	Description
db_type	VectorDBType	qdrant	The type of vector database (qdrant, chroma, etc.).
connection_string	str	http://localhost:6333	Connection string or path for the database.
api_key	Optional[SecretStr]	None	API key for the vector database service.
collection_name	str	technical_blog_posts	Name of the collection to store vectors in.
text_vector_dimension	int	1536	Dimension of text vectors, matching the embedding model.
Sources: [monitor/config.py:151-170]

ArticleProcessingConfig

Controls how individual articles are processed after discovery.

Attribute	Type	Default	Description
full_content_capture	bool	True	If true, visit the article URL to extract full content.
max_articles_per_feed	int	50	Safety cap on articles to pull per feed per run.
concurrent_article_tasks	int	5	Number of articles to process concurrently.
archive_html	bool	True	Persist the cleaned article HTML.
archive_screenshots	bool	True	Persist Playwright screenshots.
Sources: [monitor/config.py:207-240]

Usage and Loading

Application Startup

The main application entry point in monitor/main.py loads the settings and uses them to initialize an AppContext, which holds all shared components.

This sequence diagram shows how settings are loaded and passed through the application on startup.

Command-Line Overrides

The application accepts command-line arguments to override certain settings, such as the log level or running for a specific feed only. These arguments are parsed in monitor/main.py and applied to the loaded Settings object.

# monitor/main.py:455-462
parser.add_argument(
    "--log-level",
    choices=[level.value for level in LogLevel],
    default=None,
    help="Set the log level"
)

Sources: [monitor/main.py:441-462]

Configuration in Tests

For testing, configuration objects can be instantiated directly in the test code. This allows for isolated testing of components without relying on environment variables.

# test_basic.py:117-123
# Create a minimal vector database configuration
from monitor.config import VectorDBConfig, VectorDBType
vector_db_config = VectorDBConfig(
    db_type=VectorDBType.QDRANT,  # Will use in-memory client
    connection_string="http://localhost:6333",
    collection_name="test-collection",
    text_vector_dimension=len(record.text_embedding),
)

Sources: [test_basic.py:87-92, 117-123]

Debugging

To check the current configuration values as loaded by the application, you can run the following command. This is useful for verifying that .env files and environment variables are being loaded correctly.

poetry run python -c "from monitor.config import load_settings; print(load_settings())"

Sources: [WARP.md:65-66]