Research Journal & History
Relevant source files
The following files were used as context for generating this wiki page:
Scoutline maintains a durable record of investigation activities through its Artifact Store and Research Journal. This system provides reproducibility, provenance tracking, and historical recall without requiring network access or Provider credentials. The module centralizes saved report inventory and automated journaling of research findings.
Sources: packages/scoutline/src/commands/history.ts:15-35, packages/scoutline/tests/science-docs.test.js:180-200
Artifact Storage Architecture
The storage system uses a flat-file database structure located by default in ~/.scoutline/artifacts/. You can override this path using the SCOUTLINE_ARTIFACTS_DIR environment variable. The core of the store is index.json, which serves as a central log for all saved operations and journal entries.
Sources: packages/scoutline/tests/artifacts-store.test.js:68-85, packages/scoutline/src/commands/history.ts:36-40
Identity and Integrity
Every operation receives a unique requestId formatted as <UTC compact>-<4 lowercase hex>. This ID ensures lexicographical ordering by time while preventing collisions. Files are written using an atomic discipline to prevent corruption, using 0700 permissions for directories and 0600 for files to ensure local privacy.
Sources: packages/scoutline/tests/artifacts-store.test.js:25-55, packages/scoutline/tests/artifacts-store.test.js:105-120
Entry Kinds
The system distinguishes between two primary entry types:
- Save Entries: Detailed investigation results that include a log entry and a corresponding "master" report file (JSON or Markdown).
- Journal Entries: Metadata-only "skeletons" captured automatically during search, read, and research tasks. These are stored directly in the log without separate master files.
Sources: packages/scoutline/src/commands/history.ts:80-115, packages/scoutline/src/commands/history.ts:365-385
The History Command Surface
The history command provides subcommands to query, display, and manage the artifact store. It operates as a credential-free entry point that does not initialize Providers or Adapters.
Subcommand Overview
| Subcommand | Action | Mutating |
|---|---|---|
list | Displays a filtered inventory of saved runs and findings. | No |
show | Joins log metadata with the full content of a saved master report. | No |
stats | Aggregates counts by command, format, and storage size. | No |
note | Appends a manual entry to the research journal. | Yes |
recall | Performs lexical token scoring to re-find past research. | No |
export | Generates a deterministic Markdown dossier of findings. | No |
clear | Rewrites the log or deletes master files to free space. | Yes |
Sources: packages/scoutline/src/commands/history.ts:25-34, packages/scoutline/src/commands/history.ts:405-442
Show and Recovery Flow
The show operation performs a join between the log record and the physical master file. If a file was budgeted during its original run, history show recovers the full, untrimmed content from the master artifact rather than the projected output.
Research Journaling Mechanism
Journaling is an always-on feature that records investigation "skeletons" to build a searchable knowledge base. It captures queries, provider routing, and content digests.
Automated Recording
Commands like search, read, and research automatically append findings to the journal. The system uses "warm-repeat" markers to avoid redundant entries for the same query, updating the lastAsked timestamp instead of creating a new row.
Sources: packages/scoutline/src/commands/history.ts:125-145, packages/scoutline/tests/agent-registration-docs.test.js:35-50
Manual Entry (history note)
You can write explicit observations using history note. These entries are local-only and redacted at the write seam to prevent leaking secrets. Manual notes are identified by the provider sentinel note.
Sources: packages/scoutline/src/commands/history.ts:470-495, packages/scoutline/src/commands/history.ts:500-515
Retrieval and Retrieval Logic
Scoutline provides two primary ways to consume the journal: lexical recall and structured export.
Lexical Recall
The recall subcommand ranks journal entries based on token overlap between the search text and recorded queries or findings. Results are ranked by score then recency. This process is entirely local and does not use master files or network resources.
Dossier Export
history export generates a deterministic Markdown document. It inlines the body of full master reports (if saveRef is present) within the finding sections. The export uses Markdown escaping to prevent hostile log entries from breaking the dossier structure.
Sources: packages/scoutline/src/commands/history.ts:310-340, packages/scoutline/src/commands/history.ts:345-360
Lifecycle Management
The history clear command acts as a management valve for the store.
- Bare Clear: Removes only journal entries, acting as a "soft" reset for the fast-refilling discovery layer.
- Full Wipe (
--all): Deletes all log entries and unlinks all master artifact files from the disk.
The system uses write locks to ensure consistency during these mutating operations. If a log is corrupt, the clear operation fails open by initializing a new, valid empty state.
Sources: packages/scoutline/src/commands/history.ts:255-280, packages/scoutline/src/commands/history.ts:630-650
Scoutline's Research Journal and History system provides a robust local infrastructure for tracking investigation provenance and recovering full investigation states, ensuring that all AI-driven research is reproducible and auditable.