Reddit & X (Twitter) Collectors
Relevant source files
The following files were used as context for generating this wiki page:
Reddit and X (Twitter) Collectors are specialized modules used to discover and extract public social media posts. They integrate with external providers like Scavio and TinyFish to retrieve structured data for the Vantage social listening pipeline. These collectors transform raw social platform data into a normalized NewDocument format for further analysis and lead scoring.
Sources: README.md:38-41, lib/collectors/reddit.ts:16-20, lib/collectors/x.ts:16-20
Architecture Overview
The system uses a tiered provider strategy to ensure maximum coverage and cost-efficiency. It prioritizes free search and fetch methods before escalating to paid structured APIs or browser agents.
Provider Tiering Logic
- TinyFish Search + Fetch (Free): For Reddit, the system first attempts to find thread links via search and then fetches public page content as markdown.
- TinyFish Agent (Paid): If free search fails, a browser-based agent performs "strict" mode extraction of visible page text.
- Scavio API (Paid): Provides structured API access to Reddit and X data using the
SCAVIO_API_KEY. - Fixtures (Demo): If no API keys are present and the environment is not production, the system falls back to static JSON fixtures for testing.
Sources: lib/reddit/client.ts:121-140, test/social-provider-routing.test.ts:32-41, README.md:46-52
Data Flow Diagram
The following diagram illustrates how the collectors decide which provider to use based on configuration and API availability.
Reddit Collector Implementation
The Reddit collector extracts data from specific subreddits or via broader keyword queries. It identifies three distinct types of content quality: preview (discovery excerpts), page_excerpt (full thread text), and thread (structured API data).
Key Functions and Components
| Component | Description |
|---|---|
redditCollector.run | The entry point that orchestrates the collection process within the Vantage pipeline. |
fetchRedditPostsWithMeta | High-level client function that implements the provider fallback logic and input validation. |
mapPosts | Normalizes various raw record shapes (from Scavio, TinyFish, or agents) into a consistent RedditPost object. |
threadLink | A validation utility that parses and validates Reddit permalinks to ensure they point to valid public threads. |
Sources: lib/collectors/reddit.ts:14-16, lib/reddit/client.ts:36-68, lib/reddit/client.ts:121-140
Normalization Logic
The collector prefers canonical permalinks over outbound links. It truncates thread bodies to 20,000 characters and captures up to 5 top comments (limited to 4,000 characters each) to provide context for drafting.
Sources: lib/reddit/client.ts:50-65, test/social-provider-routing.test.ts:120-125
X (Twitter) Collector Implementation
The X collector primarily utilizes the Scavio x.search tool. It supports different search types, such as Latest or Top, and enforces strict validation of the returned data shape to prevent "invented" posts.
Request Sequence
The X collector follows a synchronous request-response flow with Scavio.
Validation and Security
The X client reports "provider-shape failure" if results are missing IDs or timestamps, or if URLs point to unexpected domains. It specifically enforces that URLs must be derived from supplied IDs and handles to ensure canonical reliability.
Sources: test/social-provider-routing.test.ts:169-183
Configuration and Metadata
Collectors use a standardized configuration object and return detailed metadata regarding the source of the data.
Configuration Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
query | string | Optional | Keyword search query (e.g., "AI developer tools"). |
subreddit | string | Optional | Specific subreddit to target (Reddit only). |
limit | number | No | Number of posts to retrieve (1-50, default 20). |
cursor | string | No | Pagination token for subsequent runs. |
Sources: lib/reddit/client.ts:22-26, lib/collectors/reddit.ts:18-21, lib/collectors/x.ts:18-21
Metadata Schema
Documents inserted into the database include a metadata JSONB field containing:
provider: One ofscavio,tinyfish_search_fetch,tinyfish_agent, orfixture.contentKind: Indicates if content is apreview(discoveryOnly: true) or a fullthread.score/like_count: Platform-specific engagement metrics.publicationDateMissing: Boolean flag; if true, the document is marked aspartial.
Sources: lib/collectors/reddit.ts:46-60, lib/reddit/client.ts:11-14, test/social-provider-routing.test.ts:60-65
Summary
Reddit & X Collectors serve as the primary ingress points for social signals in Vantage. They ensure data integrity by validating every URL and timestamp while managing operational costs through a multi-tier provider strategy. By normalizing these distinct platforms into a shared schema, they allow the downstream pipeline to perform unified intent scoring and draft generation.