Search & Fan-Out
Relevant source files
The following files were used as context for generating this wiki page:
The Search and Fan-Out system enables the CLI to perform parallel web searches across multiple providers simultaneously. It handles query splitting, parallel execution, deduplication of results through canonical URL identity, and ranking via fusion algorithms.
The system is designed to provide a unified search interface while abstracting provider-specific differences in credentials and transport. It manages complex workflows like query merging (using the | pipe operator) and "near-duplicate" clustering of results that share similar titles but different URLs.
Activation Tiers
The system resolves how many and which providers to use based on a hierarchy of precedence. This resolution occurs in the resolveFanoutPlan function.
- Explicit Provider Flag: Using
--provider <id,id,...>or--provider allactivates fan-out. Theallsentinel expands to every configured provider advertising the search capability in registry order. - Environment Variable Pin: If
SCOUTLINE_PROVIDERis set, the system uses a single provider and ignores fan-out configurations. - Configuration Toggle: When
scoutline config set fanout trueis active and no explicit pin exists, the system fans out to providers listed inrouting.searchor all configured search providers. - Default: If no pins or toggles are present, the system defaults to a single provider path.
Sources: packages/scoutline/src/commands/search.ts:1083-1188, packages/scoutline/tests/search-fanout.test.js:464-601
Fan-Out Logic Flow
This diagram illustrates how the system determines whether to use a single provider or initiate a fan-out.
Parallel Execution and Error Handling
The executeFanoutPlan function orchestrates parallel search requests. It creates a separate adapter instance for each "arm" (provider) to ensure transport isolation.
- Concurrency: Sub-queries and multiple provider arms are executed using
Promise.allandPromise.allSettled. - Partial Failures: If one arm fails, the system emits a stderr notice and continues with other successful arms. An
UnsupportedOptionError(e.g., a provider not supporting a specific filter) causes that specific arm to be dropped silently from results while alerting the user. - Total Failure: If every arm fails, the system propagates the error from the last attempted arm.
Sources: packages/scoutline/src/commands/search.ts:1240-1361, packages/scoutline/tests/search-fanout.test.js:655-738
Merging and Deduplication
When results return from multiple sub-queries or providers, mergeResults collapses them into a single list.
Canonical URL Identity
Results are deduped by a "canonical" version of their URL. The canonicalUrl function applies the following normalization:
- Lowercases schemes and hosts.
- Strips default ports (:80 for http, :443 for https).
- Removes URL fragments and trailing slashes.
- Removes common tracking parameters like
utm_*andfbclid(case-insensitively).
Sources: packages/scoutline/tests/search-fanout.test.js:33-149, packages/scoutline/src/commands/search.ts:983-993
Near-Duplicate Clustering
The system identifies results that are likely the same content on different URLs using title similarity.
- Shingling: Titles are split into 3-word sliding windows called shingles.
- Jaccard Similarity: If two results have a Jaccard similarity coefficient ≥ 0.80 on their shingle sets, they are clustered.
- Representative Selection: The cluster is represented by the member with the highest rank score. Other URLs are stored in the
clusterUrlsfield.
Sources: packages/scoutline/src/commands/search.ts:803-918, packages/scoutline/tests/fusion.test.js:464-585
Ranking Algorithms
Users can toggle between two ranking modes via the fusion config key or SCOUTLINE_FUSION environment variable.
| Mode | Algorithm | Description |
|---|---|---|
rrf (Default) | Reciprocal Rank Fusion | Ranks by Σ 1 / (60 + rank) across all occurrences. Produces a fusionScore. |
occurrence | Occurrence Count | Ranks by total number of sub-queries/providers that found the URL, then by best position. |
Sources: packages/scoutline/src/commands/search.ts:771-801, packages/scoutline/tests/fusion.test.js:28-118
Output and Budgeting
The system implements a "ladder" approach to fit output within a specific character limit defined by --max-chars.
Constraints and Invariants
- Never-cut fields:
url,title, andrankare preserved at every budgeting level. - Artifact Store: When budgeting triggers, the full, unbudgeted result is saved to the artifacts store for later recovery via
scoutline history show.
Sources: packages/scoutline/src/commands/search.ts:709-756, packages/scoutline/tests/search-budget.test.js:145-210
Search Controls
Search requests accept several optional controls that are mapped to provider-specific parameters.
| Option | Description | Valid Values |
|---|---|---|
domain | Limits search to a specific domain. | e.g., github.com |
recency | Filters results by publication time. | oneDay, oneWeek, oneMonth, oneYear |
contentSize | Influences result depth or extraction mode. | medium, high |
topic | Hint for query context. | general, news, finance |
type | Content type axis (Brave only). | video |
Sources: packages/scoutline/src/commands/search.ts:40-52, packages/scoutline/src/capabilities/search.ts:16-43
Summary
Search & Fan-Out provides a robust infrastructure for high-scale investigative research. By utilizing parallel execution and sophisticated fusion ranking, it allows users to compare results across multiple web indices while maintaining a clean, deduped output. The system ensures data integrity through canonical identity and provides safety mechanisms like the output budget and credential redaction during artifact storage.