Adding New Sources & Collectors
Relevant source files
The following files were used as context for generating this wiki page:Vantage uses a decoupled architecture for social listening and lead capture. Sources define the configuration and metadata for a specific feed or API, while Collectors implement the logic to fetch, normalize, and store content as documents.
Sources are categorized by their access model (lane) and type (platform). When a source is polled, the system executes the corresponding collector to transform external data into the internal document format, which then feeds the lead generation pipeline.
Sources: [README.md:12-19], [lib/db/schema.ts:107-129]
Source Architecture & Data Model
The system stores source configurations in the source table. Each source belongs to a workspace and tracks its health, polling state, and platform-specific configuration.
Data Relationships
The following diagram illustrates how sources relate to workspaces and the resulting documents and leads.
Diagram shows the one-to-many relationship between workspaces and their configured sources, and the flow of data from sources to documents.
Sources: [lib/db/schema.ts:81-180], [drizzle/0000_m1_core.sql:19-74]
Key Data Structures
| Field | Type | Description |
|---|---|---|
type | source_type | The platform identifier (e.g., hn, rss, reddit, x, github). |
lane | source_lane | Access level: free (public), paid (billed calls), or byok (user keys). |
config | jsonb | Platform-specific parameters like feedUrl, queries, or subreddit. |
health | source_health | Operational status: healthy, degraded, failing, paused, etc. |
cursor | text | Pagination state for the next poll (e.g., timestamp, page number, or API token). |
Sources: [lib/db/schema.ts:22-68], [lib/db/schema.ts:107-129]
Implementing a Collector
A Collector is an object implementing the Collector interface. It contains a run function that fetches data based on the provided CollectorContext.
The Collector Interface
Collectors must return a CollectorResult containing new documents and the updated state (etag/cursor) for the next run.
export const exampleCollector: Collector = {
name: "example",
async run(ctx: CollectorContext): Promise<CollectorResult> {
// 1. Extract config from ctx.config
// 2. Fetch data from external API (using ctx.signal for cancellation)
// 3. Normalize data into NewDocument[] array
// 4. Return documents and nextState (cursor/etag)
},
};
Sources: [lib/collectors/rss.ts:60-128], [lib/collectors/github.ts:32-117]
Content Normalization
Every document produced by a collector must include a contentHash to prevent duplicates within a workspace. The system provides a contentHash utility that combines the platform, canonical URL, and content string.
// Example from RSS collector
documents.push({
workspaceId: ctx.workspaceId,
sourceId: ctx.sourceId,
urlCanonical,
platform: "rss",
contentMd: content,
contentHash: contentHash("rss", urlCanonical, content),
// ... other fields
});
Sources: [lib/collectors/rss.ts:109-122], [lib/db/schema.ts:153-156]
Adding Sources to the System
Users add sources either by providing a custom RSS/Atom feed URL or by selecting an entry from the pre-defined community catalog.
Installation Flow
The installation process validates workspace authorization and plan limits before persisting the source to the database.
Diagram shows the sequence of checks performed in the installSource internal action.
Sources: [lib/sources/actions.ts:25-40]
Custom Feed Installation
The addFeed action allows users to add arbitrary public RSS/Atom feeds. It validates the URL length (max 2,000 chars) and ensures the URL is a public HTTP(S) link.
Sources: [lib/sources/actions.ts:98-111]
Community Catalog
The COMMUNITY_SOURCE_CATALOG contains pre-configured sources. When a user adds a source from the catalog, the system copies the config, type, and lane from the catalog entry.
| ID | Platform | Lane | Category |
|---|---|---|---|
hn-ai-coding | hn | free | Developer discussions |
github-langgraph | github | free | GitHub issues |
reddit-localllama | reddit | byok | Reddit communities |
linkedin-indexed | linkedin | paid | Paid research |
Sources: [lib/communities/catalog.ts:14-43]
Executing Scans
Sources are scanned via the scanSource (individual) or scanNow (workspace-wide) actions.
Polling Sequence
The polling logic manages concurrency using a workspace-level lease and updates the source's health based on the collector's output.
Sequence diagram illustrating the execution flow from an action request through the collector run to the final opportunity update.
Sources: [lib/sources/actions.ts:58-74]
Batch Scanning
The scanNow action triggers scanWorkspace, which iterates through all active sources. It handles partial failures, reporting if some sources failed while others succeeded, and identifies sources that are currently paused by the operator.
Sources: [lib/sources/actions.ts:77-96], [README.md:31-32]
Summary
Adding new sources involves registering platform metadata in the sourceTypeEnum and implementing a corresponding Collector that handles platform-specific API interactions and pagination cursors. The system manages these sources through a centralized action layer that enforces workspace-based multi-tenancy, plan limits, and operational health tracking. Successful collection flows directly into the document storage, enabling the downstream pipeline to generate leads and opportunities.
Sources: [lib/db/schema.ts:22-35], [lib/sources/actions.ts:25-40], [README.md:24-32]