Watch & Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
The watch system provides keyless page monitoring to track changes on specific web targets over time. It allows you to register URLs, execute periodic monitoring "ticks," and access historical change data without requiring provider credentials or complex configurations.
The system is stateful by design, persisting monitoring data under a local directory. It uses a snapshot ring buffer to store page versions and an append-only change log to record transitions such as content updates, permanent moves, or fetch errors.
Sources: packages/scoutline/src/commands/watch.ts:31-36, packages/scoutline/src/commands/watch.ts:51-54
System Architecture
The monitoring system consists of a CLI command surface, a stateful data store, and a differencing engine.
Core Components
- Watch Store: Manages the registry of monitored targets and their associated snapshots and logs.
- Snapshot Ring: A bounded buffer that stores the raw bytes and metadata of page versions. It prunes old generations based on a configurable
keeplimit (default 5). - Change Log: An append-only JSONL file for each target that records every monitoring outcome.
- Differencing Engine: Performs structural and hash-based comparisons between the current live page and the most recent snapshot.
Sources: packages/scoutline/src/lib/watch-store.ts, packages/scoutline/src/commands/watch.ts:474-486, packages/scoutline/tests/watch-store.test.js:15-30
Logic Flow
The following diagram illustrates the lifecycle of a monitoring tick for a single target:
The system uses specific locks (watch-tick-<id>) to serialize monitoring ticks and prevent race conditions between overlapping cron jobs.
Sources: packages/scoutline/src/commands/watch.ts:108-118, packages/scoutline/src/commands/watch.ts:404-450, packages/scoutline/src/commands/watch.ts:503-518
Command Interface
The watch command supports several subcommands for managing targets and monitoring their state.
Command Summary
| Subcommand | Description | Primary Options |
|---|---|---|
add | Registers a new HTTP(S) URL for monitoring. | --name, --keep |
list | Displays all registered targets and their configurations. | -O (output format) |
remove | Deletes a target from the registry. | --purge |
run | Executes a monitoring tick for one target or all targets. | --all, --timeout |
feed | Exports the change history in JSONL or RSS 2.0 format. | --format |
Sources: packages/scoutline/src/commands/watch.ts:60-76, packages/scoutline/tests/watch-command.test.js:150-160
Monitoring Outcomes (Exit Contract)
The watch run command follows a strict exit code contract to facilitate automated monitoring:
- Exit 0: No change detected or a baseline was established.
- Exit 1: Changes detected in page content or the URL has permanently moved.
- Exit 2: A fetch error occurred (e.g., timeout, 404, 500).
When running with --all, the worst exit code among all targets is returned (2 > 1 > 0).
Sources: packages/scoutline/src/commands/watch.ts:654-660, packages/scoutline/src/commands/watch.ts:684-687
Data Storage & State
All state lives under SCOUTLINE_WATCH_DIR, which defaults to ~/.scoutline/watch. The system rejects the --isolated flag because unique namespaces would orphan existing monitored state.
File Structure
WATCH_DIR/
├── targets.json # Main registry file
└── <target-id>/ # Per-target evidence directory
├── change-log.jsonl # History of all ticks
└── snapshots/ # Bounded ring of page content
├── gen-1.snapshot # Metadata (JSON)
├── gen-1.snapshot.bytes # Content (Binary)
└── ...
Sources: packages/scoutline/src/commands/watch.ts:51-54, packages/scoutline/src/lib/watch-store.ts, packages/scoutline/tests/watch-store.test.js:41-45
Registry Model
The registry is stored in targets.json and uses the following structure for each WatchTarget:
| Field | Type | Description |
|---|---|---|
id | string | Sortable, unique ID (UTC timestamp + hex tail). |
name | string | Unique, case-sensitive label (defaults to host+path slug). |
url | string | The registered HTTP(S) URL. |
keep | number | Maximum number of snapshots to retain (1-100). |
createdAt | string | ISO timestamp of registration. |
Sources: packages/scoutline/src/commands/watch.ts:121-130, packages/scoutline/tests/watch-store.test.js:82-90
Change History & Feeds
The watch feed command allows you to consume monitoring data in machine-readable formats. It redacts configured secrets from the output to prevent credential leaks.
RSS 2.0 Feed
The RSS feed provides a standardized way to consume change alerts.
- Channel Title:
scoutline watch: <target-name> - Items: Includes only
changeandmovedentries. - GUID: A unique string formatted as
<targetId>:<generation>. - Safety: The system XML-escapes section headings and strips XML-forbidden control characters.
Sources: packages/scoutline/src/commands/watch.ts:882-895, packages/scoutline/src/commands/watch.ts:909-930
JSONL Feed
The JSONL feed provides a raw, line-delimited stream of the entire change log. It is emitted with one line at a time to minimize memory usage when processing large logs.
Sources: packages/scoutline/src/commands/watch.ts:948-963, packages/scoutline/tests/watch-command.test.js:630-645
Differencing Logic
The system utilizes structural comparison for HTML content and hash-based comparison for non-HTML or binary data.
- HTML Comparison: Extracts text sections and compares them. If sections are identical, the outcome is "no-change" even if invisible parts (like comments) differ.
- Binary/Non-HTML: Compares the raw SHA-256 hashes of the content.
- URL Comparison: Uses
isSameDocumentUrlto ignore trivial differences like trailing slashes. A change in the final URL after following redirects is logged as a "moved" result if it differs from the registered URL.
Sources: packages/scoutline/src/commands/watch.ts:494-520, packages/scoutline/src/commands/watch.ts:544-555
Conclusion
Watch & Monitoring enables robust, keyless tracking of web pages through a stateful local registry. By combining a bounded snapshot ring with an append-only change log and a standardized exit contract, it provides a reliable foundation for automated research pipelines and archival verification.