Observability: Logging & Metrics

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/config.py
• monitor/main.py
• monitor/web/app.py
• WARP.md
• pyproject.toml
• README.md

The Technical Blog Monitor is equipped with a robust observability stack designed for production environments. This includes structured logging, optional Prometheus metrics, a web dashboard with health checks, and graceful shutdown handling. These features provide deep visibility into the daemon's operations, performance, and health, facilitating debugging and monitoring at scale.

The core components for observability are structlog for logging and prometheus-client for metrics, which are defined as project dependencies. The system is configured to be transparent, allowing operators to easily track feed processing, content extraction, and vector storage operations.

Sources: README.md:16, WARP.md:144, pyproject.toml:13,14

Logging

The application utilizes structlog for structured, context-aware logging. This allows log output to be easily parsed by log management systems, especially when configured for JSON output in production environments.

Sources: monitor/main.py:21, monitor/config.py:202

Configuration

Logging behavior is controlled through the MetricsConfig class in the settings. Key options allow for toggling structured logging and setting the minimum log level.

This diagram shows the structure of the MetricsConfig class.

The following table details the primary logging configuration options:

Parameter	Type	Default	Description
log_level	LogLevel (Enum)	INFO	The minimum level of logs to output.
structured_logging	bool	True	If True, logs are rendered as JSON. If False, a human-readable console format is used.

Sources: monitor/config.py:201-202

Initialization and Formatting

Logging is initialized by the setup_logging function at application startup. This function configures the structlog processing pipeline based on the structured_logging setting.

The following flowchart illustrates the logging setup process:

The setup_logging function configures a chain of structlog processors, including:

• merge_contextvars: To include context-specific data.
• add_log_level: To add the log level to the event dictionary.
• TimeStamper: To add an ISO-formatted timestamp.
• JSONRenderer or ConsoleRenderer: To format the final log entry.

# monitor/main.py:444-456
structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.processors.JSONRenderer() if settings.metrics.structured_logging
        else structlog.dev.ConsoleRenderer()
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

Sources: monitor/main.py:444-456

Metrics (Prometheus)

The application can expose internal metrics via a Prometheus-compatible endpoint. This feature is disabled by default but can be enabled via configuration for monitoring in production.

Sources: monitor/config.py:198, WARP.md:144

Configuration

Prometheus integration is managed by the MetricsConfig settings model.

Parameter	Type	Default	Description
prometheus_enabled	bool	False	If True, a Prometheus metrics server is started.
prometheus_port	int	8000	The port on which the metrics server will listen.

Sources: monitor/config.py:198-199

Implementation

The Prometheus HTTP server is started within the app_lifecycle context manager if prometheus_enabled is true. This is handled by the prometheus-client library, which is a project dependency.

This flow shows how the Prometheus server is conditionally started.

Web Dashboard and Health Checks

A lightweight web dashboard, built with FastAPI, provides an interface to view system status and recently processed articles. It includes several endpoints for observability.

Sources: monitor/web/app.py:8, monitor/dashboard.py:19

Health Check Endpoint

A dedicated /health endpoint is available for automated health checks by container orchestrators or uptime monitors.

GET /health

This endpoint returns a JSON object indicating the service status.

Field	Type	Description
status	string	Always "healthy" if the application is running.
timestamp	string	The current UTC timestamp in ISO format.
vector_db_connected	bool	True if the connection to the vector database was successful on startup.

{
  "status": "healthy",
  "timestamp": "2023-10-27T10:00:00.000Z",
  "vector_db_connected": true
}

Sources: monitor/web/app.py:198-204

Data Visibility Endpoints

The dashboard provides API endpoints to query the state of the processed data, offering another layer of observability.

• GET /api/stats: Returns aggregate statistics like total posts, posts today, and active sources.
• GET /api/posts: Returns a list of the most recently processed blog posts.

The following sequence diagram shows how the web dashboard frontend fetches data to render the page.

Graceful Shutdown

The application is designed to shut down gracefully, ensuring that in-flight tasks are completed or canceled cleanly and all resources are released. This is managed by signal handlers and a structured shutdown sequence.

Signal Handling

The setup_signal_handlers function registers handlers for SIGINT and SIGTERM. When a shutdown signal is received, it stops the main event loop, which triggers the shutdown process.

Sources: monitor/main.py:466-477

Shutdown Logic

The AppContext.shutdown method orchestrates the cleanup of all components:

1. The shutdown_event is set to signal all long-running tasks to stop.
2. All active asyncio.Task instances are canceled.
3. The APScheduler is shut down.
4. The AsyncExitStack is closed, which in turn closes the cache client, browser pool, embedding client, and vector DB client in the reverse order of their creation.
5. The ThreadPoolExecutor for CPU-bound tasks is shut down.

This ensures that file handles, network connections, and other resources are properly released, preventing resource leaks.

Sources: monitor/main.py:202-232