System Monitoring & Metrics

Relevant source files

The following files were used as context for generating this wiki page:

• tldw_Server_API/app/core/Monitoring/README.md
• tldw_Server_API/app/core/Metrics/metrics_manager.py
• Docs/Plans/2026-04-04-ftue-chat-monitoring-watchlists-review.md
• Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_05_operational_monitoring.md
• Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_09_information_architecture.md
• tldw_Server_API/app/core/Watchlists/README.md
• Docs/Architecture.md

Introduction

The System Monitoring & Metrics module in tldw_server provides real-time observability and alerting for server health, resource utilization, and content-based topic monitoring. It integrates system-level performance tracking (CPU, memory, disk) with application-specific metrics (request throughput, RAG performance, and STT/TTS latency) to ensure operational stability.

The system is architected around three primary pillars: a Metrics Registry for performance telemetry, a Topic Monitoring Service for content-based alerting, and an Operational Dashboard for administrative oversight. These components allow administrators to define threshold-based alert rules and receive notifications via multiple channels, including JSONL logs, webhooks, and email.

Monitoring Architecture & Data Flow

The monitoring system orchestrates data from various subsystems, including the API, Database, LLM providers, and Background Workers.

Component Interaction

The following diagram illustrates the relationship between telemetry collection, the central monitoring services, and the output sinks.

Topic Monitoring Service

The Topic Monitoring Service evaluates free-text strings from chat inputs/outputs, ingestion summaries, and RAG results against configured watchlists. It supports global, user, team, and org scopes for rule evaluation.

• Logic: It loads watchlist JSON, compiles regex rules, and evaluates text while deduplicating alerts within a configurable time window.
• Safety: Dangerous regex patterns are rejected, and scanned snippets are bounded to avoid large payloads.
  Sources: tldw_Server_API/app/core/Monitoring/README.md:28-35

Metrics Management

The MetricsRegistry acts as the central coordinator for all telemetry data. It utilizes normalized metric names and labels to support Prometheus-compatible exports.

Metric Types

Type	Purpose	Example
Counter	Cumulative counts that only increase	request_count, job_completions
Gauge	Values that can go up and down	cpu_usage, memory_mb, queue_depth
Histogram	Observation distributions	latency_ms, response_time
Sources: tldw_Server_API/app/core/Metrics/metrics_manager.py:20-50, Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_05_operational_monitoring.md:25-35

Registry Operations

The MetricsRegistry supports high-frequency updates using re-entrant locks (RLock) to protect in-memory data structures. Key functions include:

• register_metric: Declares a new telemetry point with associated labels.
• record / increment: Updates the current value of a metric.
• export_prometheus_format: Generates text-based telemetry for scraping.
  Sources: tldw_Server_API/app/core/Metrics/metrics_manager.py:80-150

Alerting & Notifications

Alerts are generated when system metrics cross defined thresholds or when topic rules match scanned content.

Alert Rule Structure

Administrators can configure rules via the Admin UI with the following parameters:

Parameter	Description	Typical Values
Metric	The telemetry key to monitor	cpu_usage, disk_utilization
Operator	Comparison logic	>, <, ==
Threshold	The trigger value	90%, 500ms
Duration	Minutes the state must persist	5 min, 10 min
Severity	Impact level	Low, Medium, High, Critical
Sources: Docs/Plans/2026-04-04-ftue-chat-monitoring-watchlists-review.md:75-90, Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_05_operational_monitoring.md:45-55

Notification Delivery

The NotificationService handles the dispatch of alerts through configured sinks.

1. JSONL Sink: A local file-based audit log (e.g., monitoring_notifications.log).
2. Webhook: Optional POST requests to external URLs.
3. Email: Best-effort SMTP delivery for critical alerts.
   Sources: tldw_Server_API/app/core/Monitoring/README.md:42-55

Admin Monitoring Endpoints

The FastAPI backend exposes several endpoints for managing the monitoring lifecycle.

Endpoint	Method	Description
/api/v1/monitoring/watchlists	GET	List all active watchlists.
/api/v1/monitoring/alerts	GET	Retrieve history of triggered alerts.
/api/v1/monitoring/notifications/settings	GET/PUT	Manage delivery thresholds and SMTP/Webhook config.
/api/v1/monitoring/stats	GET	Raw telemetry dump for system overview.
/api/v1/monitoring/reload	POST	Force reload of watchlists from disk.
Sources: tldw_Server_API/app/core/Monitoring/README.md:15-25, Docs/Plans/2026-04-04-ftue-chat-monitoring-watchlists-review.md:65-70

Operational Dashboard (Admin UI)

The Monitoring Dashboard provides a visual representation of system state, grouped into logical sections.

• System Overview: Visualizes CPU, Memory, Disk, and Network usage. Values are formatted into human-readable units (MB, GB, %).
• Subsystem Health: A grid showing the status of 8+ subsystems including API, Database, LLM, RAG, TTS, STT, and Embeddings.
• Recent Activity: A timeline of alert state changes, snoozed alerts, and resolved incidents.
  Sources: Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_05_operational_monitoring.md:60-75, Docs/Plans/2026-04-04-ftue-chat-monitoring-watchlists-review.md:80-95

Incident Management Workflow

When an alert triggers, the system tracks its lifecycle from discovery to resolution.

Summary

System Monitoring & Metrics in tldw_server provides a comprehensive observability stack. By combining structured metric collection via the MetricsRegistry with the regex-based TopicMonitoringService, the platform ensures that both system performance and content safety are continuously audited. The administrative interface further enables proactive incident management through configurable thresholds, automated notifications, and visual health dashboards.