Watchlists & Topic Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
- tldw_Server_API/app/core/Watchlists/README.md
- Docs/Product/Watchlists/Watchlist_PRD.md
- tldw_Server_API/app/core/Monitoring/README.md
- Docs/Product/Watchlists/Watchlists_Subscriptions_Bridge_PRD.md
- Docs/Product/Completed/Content_Collections_PRD.md
- Docs/Product/Completed/Watchlists_Subscriptions_Migration_Runbook.md
The Watchlists module is a central feature of the tldw_server platform designed for automated content aggregation, triage, and synthesis. It allows users to maintain curated collections of information sources—such as RSS feeds, websites, and news sites—and schedule recurring "jobs" to fetch, filter, and process new content. The system identifies new items since the last run using ETags, Last-Modified headers, and content hashing to ensure efficiency and avoid duplication. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:10-15, tldw_Server_API/app/core/Watchlists/README.md:3-8
Topic Monitoring acts as an alerting layer integrated with watchlists. It scans incoming text against configured watchlists to detect specific topics and keywords, emitting alerts and notifications based on severity levels. This monitoring service supports various scopes (global, user, team) and includes de-duplication logic to prevent alert fatigue. Together, these systems transform raw information streams into actionable briefings, audio "newscasts," and targeted alerts. Sources: tldw_Server_API/app/core/Monitoring/README.md:3-9, Docs/Product/Watchlists/Watchlist_PRD.md:43-48
System Architecture & Data Flow
The Watchlists system follows a pipeline-based architecture where sources are resolved into jobs, which in turn produce runs and structured outputs.
High-Level Data Flow
The run_watchlist_job orchestrator manages the lifecycle of a fetch operation. It consumes source definitions, executes fetches via RSS or site-specific rules, evaluates filters, and persists findings to the Media DB and Collections layer. Sources: tldw_Server_API/app/core/Watchlists/README.md:37-45
The diagram above illustrates the content lifecycle from source definition to final delivery. Sources: tldw_Server_API/app/core/Watchlists/README.md:37-45, Docs/Product/Watchlists/Watchlist_PRD.md:144-156
Core Components
| Component | Responsibility |
|---|---|
| Sources | Catalog of URLs with metadata (tags, groups) and fetch settings. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:89-91 |
| Scrape Jobs | Definitions of what to fetch (scope), when (schedule), and how (filters, retention). Sources: Docs/Product/Watchlists/Watchlist_PRD.md:113-116 |
| Fetchers/Parsers | Logic for HTTP GET requests with ETag support and HTML/RSS extraction. Sources: tldw_Server_API/app/core/Watchlists/README.md:41-43 |
| Topic Monitoring | Scans text against regex-based rules to emit alerts. Sources: tldw_Server_API/app/core/Monitoring/README.md:305-310 |
| Notifications | Delivery bridge for email, webhooks, and Chatbooks. Sources: tldw_Server_API/app/core/Monitoring/README.md:10-14 |
Data Models
Watchlist data is primarily stored in the user's SQLite Media DB. The system utilizes several interlinked tables to track the state of sources, jobs, and individual items.
Entity Relationship Diagram
This diagram shows the relationships between core Watchlist entities, highlighting how sources are grouped and how jobs relate to the resulting content items.
Database Table Definitions
| Table | Key Fields | Purpose |
|---|---|---|
sources | id, url, source_type, etag, last_modified | Catalog of monitored endpoints. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:162-164 |
scrape_jobs | id, scope_json, schedule_expr, output_prefs_json | Configuration for recurring tasks. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:172-174 |
scrape_runs | id, job_id, status, stats_json | Execution history and tallies of discovered items. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:175 |
scraped_items | id, media_id, url, title, status | Individual items captured during runs. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:177 |
source_seen_items | source_id, item_key, etag | Deduplication cache for RSS items. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:176 |
Job Lifecycle and Pipeline Filters
Jobs process content through a series of filters defined at the job level. These filters can include, exclude, or flag items based on keywords, regex patterns, or date ranges. Sources: Docs/Product/Watchlists/Watchlists_Subscriptions_Bridge_PRD.md:46-52
Filter Execution Sequence
The sequence shows how the filter engine determines the fate of a fetched item before persistence. Sources: Docs/Product/Watchlists/Watchlists_Subscriptions_Bridge_PRD.md:27-32, Docs/Product/Completed/Watchlists_Subscriptions_Migration_Runbook.md:104-108
Topic Monitoring & Notifications
The Topic Monitoring service evaluates text strings—such as chat inputs or ingestion summaries—against regex-based rules. It utilizes a NotificationService to dispatch alerts to various sinks. Sources: tldw_Server_API/app/core/Monitoring/README.md:3-9
Monitoring Configuration
Configuration is handled via environment variables or a monitoring_watchlists.json file.
MONITORING_WATCHLISTS_FILE: Path to regex rules. Sources: tldw_Server_API/app/core/Monitoring/README.md:30-32TOPIC_MONITOR_DEDUP_SECONDS: Window to prevent duplicate alerts (default 300s). Sources: tldw_Server_API/app/core/Monitoring/README.md:35MONITORING_NOTIFY_MIN_SEVERITY: Threshold (info, warning, critical). Sources: tldw_Server_API/app/core/Monitoring/README.md:39
Alert Scopes
Monitoring supports multiple Scopes:
- Global: System-wide rules.
- User/Team/Org: Rules scoped to specific entities. Sources: tldw_Server_API/app/core/Monitoring/README.md:5-7
API Endpoints
The system exposes REST endpoints under /api/v1/watchlists and /api/v1/monitoring.
Watchlists Endpoints
| Method | Endpoint | Description |
|---|---|---|
POST | /watchlists/sources/bulk | Bulk creation of sources with tag support. Sources: Docs/Product/Completed/Watchlists_Subscriptions_Migration_Runbook.md:52-54 |
POST | /watchlists/sources/import | Import sources from an OPML file. Sources: Docs/Product/Completed/Watchlists_Subscriptions_Migration_Runbook.md:100-102 |
POST | /watchlists/jobs/{id}/run | Manual trigger for a scrape job. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:200 |
POST | /watchlists/outputs | Generate MD/HTML briefing with delivery options. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:209-211 |
Monitoring Endpoints
| Method | Endpoint | Description |
|---|---|---|
POST | /monitoring/watchlists | Upsert a monitoring rule. Sources: tldw_Server_API/app/core/Monitoring/README.md:22 |
GET | /monitoring/alerts | List detected topic alerts. Sources: tldw_Server_API/app/core/Monitoring/README.md:26 |
POST | /monitoring/reload | Reload rules from JSON file. Sources: tldw_Server_API/app/core/Monitoring/README.md:25 |
Summary
Watchlists and Topic Monitoring provide a robust framework for automated information gathering and alerting. By combining scheduled ingest pipelines with real-time text scanning, the system ensures that users remain informed about specific topics while maintaining a structured, de-duplicated library of content in the Media DB and Collections. The integration with templates and notification services allows for the seamless transformation of raw data into highly customized briefings and critical alerts. Sources: Docs/Product/Watchlists/Watchlist_PRD.md:10-25, tldw_Server_API/app/core/Monitoring/README.md:1-15