Evidentiary Fetch
Relevant source files
The following files were used as context for generating this wiki page:
Evidentiary Fetch is a direct HTTP retrieval system used for byte-exact data preservation. It executes direct HTTP requests against target origins without mediation from AI providers or LLM translation. This system supports binary-safe downloads, cryptographic hash verification, and raw REST API interactions.
You use scoutline fetch when you need to preserve original evidence like PDFs, CSVs, or ZIP files. It prevents markdown corruption of structured data formats like JSON and XML. Unlike AI-mediated reading commands, fetch operates in a credential-free environment.
Sources: CONTEXT.md:120-123, packages/scoutline/src/commands/fetch.ts:1-12, scoutline-augmentation-spec.md:27-35
Architecture and Execution Flow
Evidentiary Fetch operates as a Direct command. The system dispatches this command before loading any credentialed provider configuration. This ensures retrieval remains independent of AI adapters or external vendor logic.
Request Execution
The executeFetch function handles the core retrieval logic. It validates the URL scheme, requiring either http:// or https://. You can specify custom HTTP methods, request bodies, and headers via CLI options.
The diagram shows the sequence from initial user command to the final processing of the response body.
Sources: packages/scoutline/src/commands/fetch.ts:60-70, packages/scoutline/src/commands/fetch.ts:153-157, packages/scoutline/src/commands/fetch.ts:212-215
Core Retrieval Logic
The system manages redirects manually to ensure safety and data integrity. It limits redirects to a maximum of 10 hops to prevent infinite loops.
Redirect Handling and Security
Fetch applies specific security policies during redirects. If a redirect points to a different origin, the system strips all caller-supplied headers. Only transport-neutral defaults like User-Agent and Accept-Encoding survive the origin jump. This prevents credential leakage to third-party servers.
| Feature | Behavior |
|---|---|
| Max Redirects | 10 Hops |
| Method Downgrade | 301/302/303 POST converts to GET |
| Cross-Origin Security | Strips non-allowlisted headers |
| Header Allowlist | user-agent, accept, accept-encoding, content-type |
Sources: packages/scoutline/src/commands/fetch.ts:221-243, packages/scoutline/tests/fetch-command.test.js:337-353
Memory Constraints
The system enforces a 50MB in-memory ceiling for all responses. This ceiling protects the CLI heap from exhaustion. You must use the --out <file> option to retrieve resources larger than 50MB. This option streams the response body directly to disk.
Sources: packages/scoutline/src/commands/fetch.ts:279-291
Evidence Preservation
Evidentiary Fetch provides tools for integrity verification. It computes checksums over the original wire bytes received from the origin.
Checksums and Integrity
You can request MD5 or SHA-256 checksums. SHA-256 serves as the collision-resistant evidentiary digest. Both digests describe the verbatim response body before any internal processing or decoding.
This diagram illustrates the streaming process that ensures byte-exact preservation and simultaneous hashing.
Sources: packages/scoutline/src/commands/fetch.ts:246-277, packages/scoutline/tests/fetch-command.test.js:145-148
Content Processing
The system classifies content as binary or text based on Content-Type and heuristics.
PDF Handling
Fetch includes specialized PDF logic. It can extract the text layer using --pdf text. If a document is structurally broken, --pdf-repair attempts to fix xref tables. Repairs and text extraction occur on separate buffers. The original wire bytes remain the primary evidence for hashing and storage.
Sources: packages/scoutline/src/commands/fetch.ts:322-332, scoutline-augmentation-spec.md:41-43
Binary vs Text Classification
The system respects binary classification to prevent "mojibake" (lossy UTF-8 decoding). If a body lacks text-like media types (e.g., text, json, xml, csv), it remains a binary payload. The stdout presentation falls back to a binary summary unless you use --raw.
Sources: packages/scoutline/src/commands/fetch.ts:365-385, packages/scoutline/tests/fetch-command.test.js:311-319
CLI Configuration and Options
The fetch command supports a variety of options for fine-tuning requests.
| Option | Description | Default |
|---|---|---|
--out <file> | Streams response to a local file | N/A |
--md5 | Reports MD5 checksum | false |
--sha256 | Reports SHA-256 evidentiary digest | false |
--ua <agent> | Overrides default browser User-Agent | Chrome/131... |
--method <verb> | Specifies HTTP method (GET, POST, etc.) | GET |
--data <@file> | Sends request body from a local file | N/A |
--timeout <ms> | Request timeout in milliseconds | 30000 |
Sources: packages/scoutline/src/commands/fetch.ts:43-55, packages/scoutline/tests/fetch-command.test.js:124-142
Prohibited Options
Fetch explicitly rejects the --max-chars option at parse time. Unlike Investigation commands, Fetch does not apply output budgeting or truncation. It failures loudly if "smuggled" budget options are detected.
Sources: packages/scoutline/src/commands/fetch.ts:162-164, packages/scoutline/tests/output-budget-t4.test.js:464-471
Summary
Evidentiary Fetch serves as the high-integrity retrieval layer for Scoutline. By bypassing AI providers and implementing strict byte preservation, it ensures that research inputs are reproducible and quotable. Its combination of manual redirect handling, cross-origin security, and cryptographic hashing makes it suitable for investigative-research pipelines requiring strong evidence chains.