Search Engine Integrations
Relevant source files
The following files were used as context for generating this wiki page:
- packages/scoutline/tests/agent-registration-docs.test.js
- packages/scoutline/tests/controls-conformance.test.js
- packages/scoutline/tests/fusion.test.js
- packages/scoutline/tests/searchapi-adapter.test.js
- packages/scoutline/tests/science-docs.test.js
- packages/scoutline/tests/context-docs.test.js
- packages/scoutline/tests/you-docs.test.js
Scoutline provides a unified interface for multiple search engine providers to facilitate agent-first web research. The system implements provider routing, result fusion, and strict control validation to ensure reproducible and comparable research results.
Sources: packages/scoutline/tests/agent-registration-docs.test.js:30-41, packages/scoutline/tests/controls-conformance.test.js:15-30
Integration Architecture
Scoutline utilizes an adapter-based architecture where each search provider is represented by a descriptor and an adapter. These adapters translate generic search requests into provider-specific API calls.
Component Flow
The search process begins at the CLI surface, moves through a validation layer, and reaches the provider-specific adapter.
This diagram shows the path of a search request from the user input to the final fused output.
Sources: packages/scoutline/tests/controls-conformance.test.js:160-200, packages/scoutline/tests/fusion.test.js:520-545
Provider Catalog
Scoutline supports various general-purpose and specialized search providers.
| Provider ID | Provider Name | Primary Capabilities |
|---|---|---|
zai | Z.AI | Search, Reader |
minimax | MiniMax | Search |
tavily | Tavily | Search, Reader, Research, Crawl, Map |
exa | Exa | Search, Reader, Research |
brave | Brave | Search |
firecrawl | Firecrawl | Search, Reader, Crawl, Map |
perplexity | Perplexity | Search, Research |
jina | Jina AI | Search, Reader, Research |
you | You.com | Search, Reader, Research |
searchapi | SearchApi.io | Search |
kagi | Kagi | Search |
Sources: packages/scoutline/tests/agent-registration-docs.test.js:80-95, packages/scoutline/tests/you-docs.test.js:115-135
Search Control Conformance
The system enforces strict conformance for search controls. Every control must either be wire-consumed by the provider or rejected during validation.
Common Controls
- Domain: Filters results to specific domains (e.g.,
--domain example.com). - Recency: Limits results by time range (e.g.,
oneDay,oneWeek,oneMonth). - ContentSize: Adjusts the depth or size of returned content (e.g.,
high). - Topic: Routes to specialized indices like
news.
Provider Conformance Table
| Provider | Domain | Recency | ContentSize | Location | Topic |
|---|---|---|---|---|---|
tavily | Consumed | Consumed | Consumed | Rejected | Consumed |
exa | Consumed | Consumed | Consumed | Consumed | Consumed |
brave | Consumed | Consumed | Consumed | Consumed | Consumed |
searchapi | Consumed | Consumed | Rejected | Consumed | Consumed |
jina | Consumed | Rejected | Rejected | Consumed | Consumed |
Sources: packages/scoutline/tests/controls-conformance.test.js:280-500, packages/scoutline/tests/searchapi-adapter.test.js:100-150
Result Fusion and Ranking
When multiple providers or sub-queries are used, Scoutline merges results through a fusion layer. The system supports two primary modes: rrf (Reciprocal Rank Fusion) and occurrence.
Fusion Mode Resolution
The system resolves the fusion mode based on a defined precedence order.
This flowchart represents the precedence logic used to determine the active ranking algorithm.
Sources: packages/scoutline/tests/fusion.test.js:115-140
Ranking Algorithms
- RRF (Reciprocal Rank Fusion): Scores results by summing
1 / (60 + rank)across all occurrences. It emits afusionScorewith three decimal places. - Occurrence: Ranks results primarily by the number of times they appear across different arms.
The tie-break chain for RRF is: Score (Desc) -> Occurrences (Desc) -> Best Position (Asc) -> First Encounter.
Sources: packages/scoutline/tests/fusion.test.js:160-220
Specialized Integrations
Science Vertical
Scoutline includes a dedicated science command for scholarly lookups. It integrates specific suppliers that ignore standard web search controls.
- Suppliers: arXiv, OpenAlex, Crossref, PubMed, Europe PMC.
- Controls: Author, Year, Venue, Type.
- Verbs:
scoutline science search <query>andscoutline science get <id>.
Sources: packages/scoutline/tests/science-docs.test.js:55-80, packages/scoutline/tests/controls-conformance.test.js:770-850
SearchApi.io
The SearchApi adapter provides a gateway to Google search engines. It maps Scoutline's news topic to the google_news engine and handles complex query parameters like site: and gl.
// SearchApi maps recency to time_period
const recencyMap = {
oneDay: "last_day",
oneWeek: "last_week",
oneMonth: "last_month",
oneYear: "last_year"
};
Sources: packages/scoutline/tests/searchapi-adapter.test.js:118-140
Configuration
Integrations require API keys configured via environment variables.
| Provider | Env Variable |
|---|---|
| Tavily | TAVILY_API_KEY |
| Exa | EXA_API_KEY |
| Brave | BRAVE_SEARCH_API_KEY |
| SearchApi | SEARCHAPI_API_KEY |
| You.com | YDC_API_KEY |
| Kagi | KAGI_API_KEY |
Sources: packages/scoutline/tests/controls-conformance.test.js:230-260, packages/scoutline/tests/you-docs.test.js:75-85
Search Engine Integrations in Scoutline provide a standardized research capability by abstracting diverse provider APIs into a single, fused result set with consistent behavioral controls.