ProductHunt Collector

Relevant source files

The following files were used as context for generating this wiki page:

• lib/collectors/producthunt.ts
• lib/producthunt/client.ts
• test/tinyfish-collectors.test.ts
• README.md
• lib/db/schema.ts
• lib/communities/catalog.ts
• drizzle/0000_m1_core.sql

The ProductHunt Collector discovers and ingests product launch data from Product Hunt into the Vantage platform. It serves as a discovery feed for identifying new tools and potential market opportunities. The system employs a multi-tiered fetching strategy to ensure data availability, prioritizing modern search and scraping techniques over standard API calls.

System Architecture

The collector follows a hierarchical retrieval pattern. It first attempts to use the TinyFish ecosystem for search and content extraction. If TinyFish is unavailable or fails, it falls back to the native Product Hunt GraphQL API, and finally to local fixtures for development and demonstration purposes.

Fetching Priority and Providers

The client attempts providers in the following order:

1. TinyFish Search + Fetch: Performs a targeted search for specific post URLs and then scrapes the markdown content.
2. TinyFish Homepage Fetch: Scrapes the Product Hunt homepage to extract featured launch cards.
3. TinyFish Agent: Runs a structured browser agent to navigate and extract data when simpler scraping fails.
4. Product Hunt API: Uses the official GraphQL API (requires PH_DEV_TOKEN).
5. Fixture: Loads pre-defined static data from local files.

Sources: lib/producthunt/client.ts:241-274, README.md:41-45

Core Logic Flow

This diagram illustrates the decision logic when the collector attempts to retrieve posts.

Data Model and Mapping

The collector transforms raw data from various providers into a unified ProductHuntPost structure before persisting it as a document in the database.

ProductHuntPost Structure

Field	Type	Description
id	string	Unique identifier (slug or product ID)
name	string	The name of the product
tagline	string	A short description or headline
url	string	Canonical Product Hunt URL
website	string	The product's own homepage URL
votesCount	number	Number of upvotes received
topics	string[]	Associated tags or categories
maker	string	Username of the creator

Sources: lib/producthunt/client.ts:16-27

Database Normalization

The productHuntCollector maps the internal client representation to the document table schema.

The transformation logic performs the following mappings:

• platform: Always set to "producthunt".
• contentMd: Constructed as a Markdown string containing the name, tagline, and full description.
• contentHash: A unique hash derived from the platform, URL, and body content to prevent duplicates.
• metadata: Stores provider information, vote counts, topics, and the website URL.

Sources: lib/collectors/producthunt.ts:21-43, lib/db/schema.ts:137-164

Implementation Details

Collector Configuration

The collector is registered in the system and accepts specific configuration parameters through the CollectorContext.

Parameter	Type	Default	Description
first	number	20	Number of posts to retrieve
query	string	undefined	Optional search query for TinyFish search

Sources: lib/collectors/producthunt.ts:14-20, lib/communities/catalog.ts:77-77

Data Extraction Logic

The system uses multiple utility functions to parse unstructured data from different sources:

• mapPosts: Normalizes object arrays from the TinyFish Agent or generic JSON rows, mapping varying field names like votes_count, upvotes, and snippet to standard fields.
• postsFromSearchAndPages: Extracts post links and details from Markdown text and search hit snippets, using regex to identify valid /posts/ paths.
• mapTopics: Sanitizes tag arrays, handling both string arrays and object arrays containing name/label fields.

Sources: lib/producthunt/client.ts:69-166

API Integration

When using the official API, the collector performs a POST request to https://api.producthunt.com/v2/api/graphql. It requests posts ordered by VOTES and retrieves the node's name, tagline, url, website, votesCount, and topics.

Sources: lib/producthunt/client.ts:200-235

Testing and Validation

Tests verify the provider priority and metadata accuracy. Specifically, the test suite ensures that:

1. TinyFish Search+Fetch is preferred over the Agent.
2. Metadata correctly identifies the provider (e.g., tinyfish_search_fetch).
3. The mocked flag is correctly set to true when falling back to fixtures.

Sources: test/tinyfish-collectors.test.ts:150-184

Summary

The ProductHunt Collector is a robust discovery module that uses a tiered retrieval approach. By prioritizing scraping and search via TinyFish before falling back to official APIs and local fixtures, it maintains high availability for product launch data while populating the Vantage platform's document store with rich, normalized metadata.