Web Reader
Relevant source files
The following files were used as context for generating this wiki page:
Web Reader fetches and parses web page content through various provider capabilities. It normalizes external web content into structured Markdown or plain text, supporting advanced features like data extraction, output budgeting, and automated URL rewriting for specific domains like GitHub Gists.
The system utilizes a provider-neutral request architecture where the core handler manages validation and projection while individual adapters handle transport and provider-specific logic. This allows Scoutline to route "read" operations through multiple suppliers such as Z.AI, Tavily, Exa, Firecrawl, and Jina AI.
Sources: packages/scoutline/src/commands/read.ts:1-35, packages/scoutline/src/commands/read.ts:800-850
Architecture and Data Flow
Web Reader follows a structured pipeline that separates input validation, provider execution, and result projection.
Command Execution Flow
The read command first performs parse-level validation on the URL scheme and extraction modes. If valid, it constructs a provider-neutral ReaderFetchRequest. This request is passed to the execution layer, which handles caching and retries before delegating to the specific provider adapter.
System Components
| Component | Responsibility |
|---|---|
read.ts Handler | Manages CLI arguments, initial validation, and final projection. |
ReaderCapability | Defines the interface for fetching web content across providers. |
ExecutionDependencies | Provides shared services like caching and sleep/retry logic. |
Adapter | Handles provider-specific wire protocols and normalization. |
Output Budget Ladder | Trims content to fit within a specific character limit. |
Sources: packages/scoutline/src/commands/read.ts:40-80, packages/scoutline/src/capabilities/reader.ts:1-20
Content Extraction Modes
Web Reader supports two primary output shapes: Content Read (default) and Extract Read.
Content Read
The default mode returns the full page body as Markdown or text. It includes metadata such as the page title, the original URL, and the final URL (useful if redirects occurred or if the URL was rewritten).
Extract Read
Activated by the --extract flag, this mode pulls specific data slices from the content.
| Mode | Description |
|---|---|
code | Extracts all fenced code blocks with their detected languages. |
links | Extracts all hyperlinks and their anchor text. |
tables | Extracts structured data from HTML tables. |
headings | Extracts the document structure (H1-H6) as a section outline. |
Sources: packages/scoutline/src/commands/read.ts:850-880, packages/scoutline/tests/reader-command.test.js:630-670
Output Budgeting (--max-chars)
Web Reader implements a "ladder" system to shrink outputs when they exceed a user-defined character budget.
Character Budgeting Sequence
When --max-chars is provided, the system applies a sequence of LadderRule operations to the resulting envelope.
The system uses specific guards to prevent content corruption during budgeting:
- Fence-awareness: Fenced code blocks are never trimmed to avoid breaking syntax.
- Heading-awareness: Heading lines are never cut mid-value; sections are dropped in their entirety instead.
- Omission Markers: Trimmed lines are prefixed with
…to indicate missing content.
Sources: packages/scoutline/src/commands/read.ts:190-280, packages/scoutline/tests/output-budget-t4.test.js:120-180
Provider Specifics
Providers advertise the reader capability with varying levels of support for Z.AI-specific options.
| Provider | Reader Implementation | Notes |
|---|---|---|
| Z.AI | Native Adapter | Full support for all flags; handles Gists. |
| Tavily | Extract Endpoint | Rejects --with-links and --no-gfm. |
| Firecrawl | /v1/scrape | Normalizes metadata; pins proxy:basic. |
| Jina AI | r.jina.ai | Supports keyless access. |
| Linkup | /fetch endpoint | Renders JavaScript by default. |
Sources: packages/scoutline/src/commands/read.ts:800-840, packages/scoutline/tests/firecrawl-adapter.test.js:150-180
URL Rewriting
The Z.AI Adapter automatically rewrites gist.github.com URLs to their /raw counterparts. This ensures the reader retrieves pure file content rather than the GitHub UI wrapper.
Sources: packages/scoutline/src/commands/read.ts:795-805
Data Models
ReaderFetchRequest
The internal structure used to communicate requirements to Provider Adapters.
export interface ReaderFetchRequest {
url: string;
format?: "markdown" | "text";
retainImages?: boolean;
withLinksSummary?: boolean;
noGfm?: boolean;
keepImgDataUrl?: boolean;
withImagesSummary?: boolean;
timeout?: number;
}
Sources: packages/scoutline/src/commands/read.ts:110-130
Content Envelope (v1)
The structured JSON object returned for standard content reads.
| Field | Type | Description |
|---|---|---|
schemaVersion | number | Always 1 for current implementation. |
url | string | The input URL. |
finalUrl | string | The actual URL fetched by the adapter. |
content | string | The normalized page body. |
truncated | boolean | Indicates if output was cut by a budget. |
originalContentLength | number | The length of content before budgeting. |
Sources: packages/scoutline/src/commands/read.ts:140-165, packages/scoutline/src/commands/read.ts:850-865
Conclusion
Web Reader provides a robust, provider-agnostic way to consume web content for AI agents and investigative research. By separating the transport layer (Adapters) from the presentation and budgeting logic (Handler), it maintains consistency across multiple suppliers while offering advanced control over data extraction and output size.