Collectors Overview

Relevant source files

The following files were used as context for generating this wiki page:

• lib/collectors/types.ts
• lib/collectors/github.ts
• lib/collectors/reddit.ts
• lib/collectors/rss.ts
• lib/collectors/alexandria.ts
• lib/collectors/linkedin.ts
• README.md
• test/source-coverage.test.ts
• test/collector-runner.test.ts

Collectors fetch and normalize content from social platforms and developer communities to support lead capture and research. They transform diverse data formats into a standardized document schema stored in the Vantage database. The system supports multiple collection methods, including public APIs, specialized scraping providers like Scavio, and RSS feeds.

Sources: README.md:14-25, lib/collectors/types.ts:1-12

Core Architecture

The collector system relies on a standard interface called Collector. Each specific implementation must define a run method that accepts a CollectorContext and returns a CollectorResult.

Key Data Structures

Type	Description	Key Fields
CollectorContext	Input parameters for a collector run	workspaceId, sourceId, config, etag, cursor
CollectorResult	Output containing normalized data and state	documents, partial, coverageReason, nextState
NewDocument	The normalized document format for persistence	urlCanonical, platform, contentMd, contentHash

Sources: lib/collectors/types.ts:13-50

Collector Execution Flow

The runner orchestrates the collection process by checking source health and global switches before invoking the specific collector logic.

This diagram illustrates the lifecycle of a collection task from initial validation to database persistence.
Sources: test/collector-runner.test.ts:74-135

Implementation Varieties

Vantage implements collectors tailored to specific platform characteristics and available APIs.

Social and Community Collectors

• Reddit: Uses the Scavio reddit.search API to find posts within specific subreddits. It prioritizes actual content over previews and tracks comment counts. Sources: lib/collectors/reddit.ts:13-35, README.md:37-37
• GitHub: Performs searches for public issues using the GitHub API. It handles complex pagination, rate limiting (403/429), and requires a query between 1-500 characters. Sources: lib/collectors/github.ts:38-100
• RSS/Substack: Fetches XML feeds and uses conditional GET (Etags/Last-Modified) to avoid redundant downloads. It parses entries into Markdown content. Sources: lib/collectors/rss.ts:47-90

Discovery and Paid Collectors

• Alexandria (Firecrawl): Accesses specialized developer indexes and repository issue datasets. It supports searching for specific repository issues or developer-focused web passages. Sources: lib/collectors/alexandria.ts:11-40
• LinkedIn: Utilizes Scavio to fetch public indexed snippets via Google search. This method avoids authenticated sessions while providing public post summaries. Sources: lib/collectors/linkedin.ts:40-65

State and Coverage Tracking

Collectors maintain state between runs to ensure efficiency and data integrity.

Pagination and Resumption

Collectors use a cursor or nextState object to track progress. For example, the GitHub collector stores since, until, and nextPage in a JSON string within the cursor to resume interrupted scans or handle rate limits.
Sources: lib/collectors/github.ts:46-60

Health Classification

The system classifies the success of a collection run based on the results and errors encountered.

Health Status	Triggering Condition
healthy	Successful run with live documents
degraded	Fixture-only data or partial/incomplete scan
access_pending	Missing API keys
budget_limited	Rate limits (429) or budget exhaustion
blocked	WAF blocks (e.g., Cloudflare)
failed	Generic execution errors

Sources: test/source-coverage.test.ts:13-60

Content Normalization

Regardless of the source, all collectors must produce a NewDocument that includes:

• urlCanonical: The unique, permanent URL for the content.
• contentHash: A unique hash derived from the platform, URL, and content to prevent duplicates.
• metadata: Platform-specific data such as subreddit, issueId, or provider info.

Sources: lib/collectors/reddit.ts:50-70, lib/collectors/github.ts:110-120

Example: GitHub Document Mapping

documents.push({
  workspaceId: ctx.workspaceId,
  sourceId: ctx.sourceId,
  platform: "github",
  urlCanonical,
  title: title || null,
  contentMd,
  postedAt: parseValidDate(issue.created_at),
  contentHash: contentHash("github", urlCanonical, contentMd),
  metadata: { 
    provider: "github", 
    issueId: issue.id, 
    labels: issue.labels.map(l => l.name) 
  }
});

Sources: lib/collectors/github.ts:110-120

Conclusion

The collector module provides a robust framework for aggregating technical and social data. By abstracting platform-specific logic into separate implementations while enforcing a common interface and health reporting system, Vantage maintains a consistent pipeline for processing external evidence.