Archival Intelligence
Relevant source files
The following files were used as context for generating this wiki page:
Archival Intelligence retrieves historical web states and temporal data for investigative research. It focuses on recovering content from restructured or offline websites using the Internet Archive.
Scoutline provides these capabilities through the archive command namespace. Users employ this system to verify page changes over time or bypass modern client-side paywalls by accessing server-rendered originals stored in archives.
Sources: scoutline-augmentation-spec.md:12-25, packages/scoutline/tests/agent-registration-docs.test.js:44-48
Core Archival Capabilities
The archival system operates through two primary verbs: cdx and get.
Enumeration via CDX
The cdx command lists Internet Archive captures for a specific URL or pattern. It produces structured JSON output containing metadata for each capture. If you provide a URL pattern, the system discovers all indexed snapshots within that scope.
Sources: scoutline-augmentation-spec.md:15-18
| Field | Type | Description |
|---|---|---|
timestamp | String | The UTC timestamp of the capture. |
statuscode | Integer | The HTTP status code returned during capture (e.g., 200). |
length | Integer | The byte length of the captured content. |
digest | String | The content hash/digest for integrity verification. |
url | String | The original source URL. |
| Sources: scoutline-augmentation-spec.md:17-18 |
Content Retrieval
The get command fetches the raw original content of a specific capture. It removes archive toolbars or injected scripts to ensure byte-faithful reproduction. If no timestamp is specified, the system automatically selects the latest or "best" capture available.
Sources: scoutline-augmentation-spec.md:21-25
The diagram shows the logic flow from user request to the specific Wayback Machine API endpoints based on the chosen command.
Sources: scoutline-augmentation-spec.md:15-25
Temporal Search Configuration
Archival searches support filtering and specific targeting through command-line flags. These options allow researchers to narrow the temporal window for page recovery.
Sources: scoutline-augmentation-spec.md:16-22
Archive Command Options
--from <timestamp>: Filters captures starting from a specific point in time.--to <timestamp>: Limits captures up to a specific date.--status <code|pattern>: Matches captures with specific HTTP status codes (e.g.,--status 200).--at <timestamp|best>: Targets a specific snapshot for retrieval.--raw: Requests the data without any processing or markdown conversion.
Sources: scoutline-augmentation-spec.md:16-22
Data Integrity and Artifacts
The system ensures reproducibility and integrity through strict artifact management. Every archival lookup generates a unique requestId based on an injected UTC timestamp and a hex tail.
Sources: packages/scoutline/tests/artifacts-store.test.js:29-57
Artifact Storage
When the system retrieves data, it performs an atomic write to the artifacts directory. This prevents partial writes or file corruption during concurrent operations. If an artifact with the same ID already exists, the system refuses to overwrite it unless you use the force option.
Sources: packages/scoutline/tests/artifacts-store.test.js:101-140
This sequence diagram illustrates the atomic write discipline used to protect archival data integrity.
Sources: packages/scoutline/tests/artifacts-store.test.js:139-190
Output Budgeting
Archival results are subject to output budgets to prevent overwhelming agent context windows. The system uses a priority ladder to decide which fields to trim or drop when the data exceeds the --max-chars limit.
Sources: packages/scoutline/tests/output-budget-t4.test.js:13-33
- Never-cut fields: The system always preserves URLs, titles, and top-level headings.
- Trim-early fields: Paragraphs and bodies are halved or truncated from the end of the document forward.
- Drop-late fields: Deep sections or non-essential metadata are removed only if trimming fails to meet the budget.
Sources: packages/scoutline/tests/output-budget-t4.test.js:115-180
Summary: Archival Intelligence provides a temporal lookup engine within scoutline. It leverages cdx enumeration and raw get retrieval to recover historical web data. The system maintains strict data integrity via atomic artifact storage and manages data volume through prioritized output budgeting.