Scraping & Agent Providers
Relevant source files
The following files were used as context for generating this wiki page:
Vantage uses a multi-tiered provider system to collect data from social platforms, developer communities, and the open web. The system prioritizes structured API access and lightweight scraping over expensive browser-based agents to maintain efficiency and budget control.
This module manages the lifecycle of data acquisition, including provider selection, fallback logic, and cost metering for paid services like Scavio, TinyFish, and Alexandria.
Provider Orchestration & Fallback Logic
Vantage implements a hierarchical fallback strategy for content collection. It attempts to resolve requests using the most cost-effective and reliable method first, escalating to complex agents or paid search only when necessary.
Primary Fallback Orders
The system defines specific orders for high-traffic platforms to balance accuracy and cost:
| Platform | Tier 1 (Preferred) | Tier 2 | Tier 3 | Tier 4 | Tier 5 |
|---|---|---|---|---|---|
| TinyFish Search+Fetch | TinyFish Agent | Scavio | — | Fixture | |
| YouTube | Scavio Comments | TinyFish Fetch | TinyFish Agent | Data API | Fixture |
| Product Hunt | TinyFish Search+Fetch | Homepage Fetch | TinyFish Agent | PH Dev Token | Fixture |
| X (Twitter) | Scavio Search | — | — | — | Fixture |
Sources: README.md:46-52, lib/reddit/client.ts:140-165
Collection Flow Diagram
The following diagram illustrates the decision logic when fetching content from a platform like Reddit:
TinyFish: Agents & Web Discovery
TinyFish serves two primary roles: discovering public links via search and extracting structured data using AI-driven browser agents.
Structured Agent Execution
The runTinyFishStructuredAgent function executes browser tasks with a specific goal and output schema. This is used for extracting exact title, body, and comment data from complex pages where standard DOM parsing fails.
- Mode: Strict
- Limits: 12 steps or 60 seconds duration
- Schema: Requires specific JSON fields (e.g.,
posts,topComments)
Sources: lib/reddit/client.ts:110-125, test/tinyfish-collectors.test.ts:50-65
Search and Fetch
Lightweight discovery uses tinyFishSearch to find URLs and tinyFishFetchMarkdown to retrieve page content as Markdown. The system tags results with contentKind to distinguish between a "preview" (search snippet) and a "page_excerpt" (full fetched text).
Sources: lib/reddit/client.ts:98-108, test/social-provider-routing.test.ts:40-55
Scavio: Social & Indexed Search
Scavio provides specialized APIs for social platforms and indexed web search. It is the primary provider for X and a fallback for YouTube and Reddit.
Capabilities and Endpoints
- X/Reddit Search: Accesses real-time social posts via the
scavioSDK. - YouTube Scrape: Directly retrieves video comments.
- LinkedIn Snippets: Uses the
/api/v2/googleendpoint to find public LinkedIn posts through indexed search, avoiding authentication requirements.
Sources: lib/collectors/linkedin.ts:30-45, test/scavio-social-collectors.test.ts:16-25, test/tinyfish-collectors.test.ts:80-95
Alexandria (Firecrawl)
Alexandria provides paid developer discovery through specific datasets, primarily indexed via Firecrawl.
Dataset Capabilities
- Firecrawl Developer Index: Searches for indexed developer passages and documentation.
- GitHub Repository Issues: A high-throughput dataset for collecting GitHub issues without hitting standard API rate limits.
Sources: lib/communities/catalog.ts:60-65, test/community-provider-collectors.test.ts:25-40
Cost Management and Paid Call Guard
Every request to a paid provider (Scavio, TinyFish Agent, Alexandria) passes through a runPaidCall wrapper. This utility ensures that operations stay within workspace budget constraints.
Metering Logic
- Estimate: Before execution, the system provides a
estimateUsdbased on the provider and action. - Execution: The provider call runs within a budget-checked context.
- Receipt: The system records the final
costUsdreturned by the provider receipt. - Denial: If a workspace has exhausted its budget,
runPaidCallthrows aPaidCallDeniedError, preventing the external HTTP request.
Sources: lib/reddit/client.ts:128-135, test/social-provider-routing.test.ts:135-145, test/community-provider-collectors.test.ts:45-55
Provider Cost Configuration
| Provider | Action | Cost Metric |
|---|---|---|
| TinyFish | Agent Step | $0.016 per step |
| Scavio | Google Search | Configurable via SCAVIO_GOOGLE_COST_USD |
| Alexandria | Dataset Search | Credit-based (e.g., 5 credits for issues) |
Sources: lib/reddit/client.ts:25, lib/collectors/linkedin.ts:32, test/community-provider-collectors.test.ts:70-75
Document Metadata & Provenance
When documents are collected, the provider identity and acquisition method are preserved in the metadata object. This allows the pipeline to judge the quality and depth of the data.
- Mocked Data: Documents originating from fixtures are marked with
mocked: trueandprovider: "fixture". - Discovery Evidence: Snippets from search results are tagged with
discoveryOnly: trueandcontentKind: "preview". - Provenance Tags: Metadata includes tags like
tinyfish_search_fetch,scavio, ortinyfish_agentto track the origin of the text.
Sources: README.md:54, test/social-provider-routing.test.ts:60-75, test/source-coverage.test.ts:10-25
The scraping and agent provider system ensures Vantage maintains high data coverage while strictly controlling operational costs through intelligent routing and budget enforcement.