Crawl & Map

Relevant source files

The following files were used as context for generating this wiki page:

• packages/scoutline/src/commands/crawl.ts
• packages/scoutline/src/commands/map.ts
• packages/scoutline/src/providers/firecrawl/adapter.ts
• packages/scoutline/src/providers/spider/adapter.ts
• packages/scoutline/tests/output-budget-t4.test.js

Crawl and Map are provider-backed capabilities that discover and retrieve website structures. The Crawl system recursively visits pages to extract content, while the Map system focuses exclusively on discovering the URL hierarchy of a target site.

System Architecture

The Crawl and Map systems follow a tiered architecture where thin command handlers delegate to provider-specific adapters through a generic execution wrapper. This wrapper manages caching and ensures that operations remain performant and cost-effective.

The architecture separates request validation, cache identity management, and output projection. Command handlers own the public envelope projection and output-mode presentation, while adapters handle transport, raw response parsing, and error normalization.

Sources: packages/scoutline/src/commands/crawl.ts:16-24, packages/scoutline/src/commands/map.ts:14-22

Crawl Capability

The Crawl command extracts page content starting from a seed URL by following links up to a defined depth and breadth. It supports filtering through path inclusion or exclusion patterns and natural language instructions.

Crawl Request Parameters

Option	Type	Description
depth	number	Crawl depth, 1–5 (Default: 1)
breadth	number	Max links to follow per page, 1–500 (Default: 20)
limit	number	Total pages to process (Default: 50)
selectPaths	string	Regex patterns to select URL paths
excludePaths	string	Regex patterns to exclude URL paths
instructions	string	Natural language instructions for page selection
format	markdown | text	Output format (Default: markdown)
contentSize	medium | high	Extraction depth (Default: medium)

Sources: packages/scoutline/src/commands/crawl.ts:37-49, packages/scoutline/src/commands/crawl.ts:168-185

Async Crawl Flow (Firecrawl)

Providers like Firecrawl implement an asynchronous lifecycle involving job creation, status polling, and eventual result collection. The system includes a "reclaim-on-miss" logic to adopt in-flight jobs if a previous process crashed or was interrupted.

Map Capability

The Map command discovers the URL structure of a website without fetching page bodies. It is primarily used for site discovery before deciding which specific pages to crawl or read.

Provider Behavior Comparison

Different providers handle mapping with varying levels of synchronicity and feature support.

Feature	Firecrawl	Spider.cloud	Tavily
Sync/Async	Synchronous (Discovery)	Synchronous One-shot	Synchronous
Deduplication	Done by API	Done via Set in Adapter	Done by API
Filter Support	Instructions (search)	limit only	limit, depth, breadth, paths

Spider.cloud specifically discovery URLs via the /links endpoint and ensures that the root mapped page is included in the final set, even if the API response omits it.

Sources: packages/scoutline/src/providers/spider/adapter.ts:246-270, packages/scoutline/src/providers/firecrawl/adapter.ts:462-480, packages/scoutline/src/commands/map.ts:145-155

Output Budgeting (Ladder Rules)

For Crawl results, Scoutline applies an "Output Budget Ladder" to ensure results fit within the character limit specified by --max-chars. This is critical for preventing oversized payloads from overwhelming downstream LLM contexts or CLI displays.

Crawl Ladder Priorities

1. Trim Page Contents: The engine performs a backward scan to halve the content of the last trimmable page. This process repeats until the budget is met or pages cannot be shrunk further.
2. Drop Trailing Pages: If trimming content is insufficient, the engine drops whole pages from the end of the results, recomputing the totalPages count to maintain accuracy.

Data Models

Crawl Result Schema

The crawl output follows schema-version-1 and provides metadata about the source and truncation state of each page.

{
  "schemaVersion": 1,
  "baseUrl": "https://example.com",
  "pages": [
    {
      "url": "https://example.com/subpage",
      "content": "... page body ...",
      "contentFormat": "markdown",
      "truncated": true,
      "originalContentLength": 5000
    }
  ],
  "totalPages": 1
}

Sources: packages/scoutline/src/commands/crawl.ts:200-215

Map Result Schema

Map results return a simple list of discovered URLs.

{
  "schemaVersion": 1,
  "baseUrl": "https://example.com",
  "urls": ["https://example.com/1", "https://example.com/2"],
  "totalUrls": 2
}

Sources: packages/scoutline/src/commands/map.ts:133-138

Summary

The Crawl and Map systems provide robust website exploration tools with deep provider integration. By utilizing asynchronous job management for large crawls and synchronous discovery for mapping, Scoutline balances reliability with responsiveness. The integrated output budgeting system ensures that even massive crawls are projected into usable, context-aware summaries for users and agents.