Hacker News Collector

Relevant source files

The following files were used as context for generating this wiki page:

• lib/collectors/hn.ts
• lib/communities/catalog.ts
• lib/collectors/types.ts
• README.md
• drizzle/0000_m1_core.sql
• test/social-provider-routing.test.ts
• lib/drafting/contribution.ts

The Hacker News (HN) Collector identifies and retrieves developer discussions from Hacker News to support lead capture and research. It queries the Algolia HN Search API to find stories based on specific keyword configurations and normalizes them into the project's standard document format. Sources: lib/collectors/hn.ts:1-7, README.md:32

This collector operates within the "free" lane of the Vantage platform, meaning it does not require user-provided API keys or paid credits to function. The retrieved documents are used by the Pipeline to score opportunities and generate research briefs. Sources: lib/communities/catalog.ts:13-14, README.md:10

Architecture and Data Flow

The HN Collector implements the Collector interface, which defines a standard run method for all content ingestion workers. It uses a time-based cursor to ensure it only retrieves new content since the last poll. Sources: lib/collectors/hn.ts:41, lib/collectors/types.ts:32-35

Data Flow Diagram

The following diagram illustrates how the collector processes a request to fetch Hacker News stories.

The collector iterates through configured search queries and fetches results from Algolia, maintaining a cursor to prevent duplicate processing in future runs. Sources: lib/collectors/hn.ts:43-80

Implementation Details

API Interaction

The collector communicates with the Algolia HN API endpoint at https://hn.algolia.com/api/v1/search_by_date. It applies numericFilters to restrict results to the window between the last successful run and the current time. Sources: lib/collectors/hn.ts:7, lib/collectors/hn.ts:49

Retrieval Constraints

To maintain performance and avoid hitting API limits, the collector applies several constraints:

• Query Limit: Processes at most 5 queries per run. Sources: lib/collectors/hn.ts:44
• Page Limit: Fetches at most 2 pages per query (default 1). Sources: lib/collectors/hn.ts:45
• Timeout: Each HTTP request to Algolia has a strict 4-second timeout. Sources: lib/collectors/hn.ts:37

Document Normalization

Raw hits from Algolia are mapped to the NewDocument schema. If a story lacks a direct URL, the collector constructs a canonical HN item link using the objectID. Sources: lib/collectors/hn.ts:84-88

Field	Source / Transformation
platform	Hardcoded to "hn"
urlCanonical	hit.url or https://news.ycombinator.com/item?id={id}
postedAt	Parsed from created_at_i (unix timestamp) or created_at (ISO string)
contentMd	story_text OR comment_text OR title
contentHash	Generated via contentHash("hn", url, body)
Sources: lib/collectors/hn.ts:92-105

Configuration Options

The collector behavior is defined by the config object stored in the source table. Sources: drizzle/0000_m1_core.sql:28

Configuration Schema

Parameter	Type	Default	Description
queries	string[]	["Show HN"]	List of search terms to track.
query	string	N/A	Single fallback query if queries is missing.
maxPages	number	1	Maximum pages to fetch per query (capped at 2).
lookbackHours	number	24	Initial hours to search if no cursor exists.
Sources: lib/collectors/hn.ts:24-30, lib/collectors/hn.ts:45-47

Example Catalog Entry

The COMMUNITY_SOURCE_CATALOG contains pre-defined HN sources, such as AI coding assistant research:

{ 
  id: "hn-ai-coding", 
  type: "hn", 
  config: { 
    queries: ["AI coding", "coding agent"], 
    maxPages: 1 
  }
}

Sources: lib/communities/catalog.ts:13

Policy and Drafting Constraints

Hacker News content is subject to specific venue rules within the Vantage platform. While content can be collected, the system explicitly prohibits generating AI-written contributions for this platform. Sources: lib/drafting/rules.ts, lib/drafting/contribution.ts:88

• Contribution Mode: Prohibited for HN. Sources: test/contribution-quality.test.ts:28
• Output: The system generates a "Research brief" instead of a draft when HN documents are used as evidence. Sources: lib/drafting/contribution.ts:88, app/opportunities/[id]/draft-panel.tsx:142

Summary

The Hacker News Collector provides a reliable mechanism for ingesting developer discussions using the Algolia search index. By enforcing strict timeouts, pagination limits, and deduplication via objectID, it efficiently populates the Vantage database with relevant threads while adhering to the platform's non-promotional research-first philosophy for HN. Sources: lib/collectors/hn.ts:43-112, lib/drafting/contribution.ts:88