YouTube Collector
Relevant source files
The following files were used as context for generating this wiki page:
The YouTube Collector gathers public comments from YouTube videos to identify relevant discussions and potential opportunities. It retrieves data using specific video identifiers or keyword search queries. The system transforms raw comment data into standardized documents within the project's database.
This collector operates within the broader ingestion pipeline. It supports multiple data acquisition paths, including third-party scraping services, official APIs, and AI-driven browser agents.
Sources: lib/collectors/youtube.ts:12-14, lib/youtube/client.ts:380-388
Architecture and Data Flow
The YouTube Collector follows a tiered acquisition strategy to ensure reliability. It attempts to fetch data from the most efficient source first, falling back to more complex or expensive methods if necessary.
Acquisition Tier Hierarchy
The system evaluates providers in the following strict order:
- Scavio API: Executes structured scrapes of YouTube comment threads.
- TinyFish Fetch: Extracts content by fetching video pages and converting them to Markdown.
- TinyFish Search: Discovers videos based on queries before fetching comments.
- TinyFish Agent: Runs a structured AI browser agent to extract comments as a last resort.
- YouTube Data API: Uses the official v3 API if developer keys are present.
- Fixtures: Loads static mock data for testing or demonstration purposes.
Sources: lib/youtube/client.ts:380-435, test/tinyfish-collectors.test.ts:98-150
Collection Sequence
The following diagram illustrates the sequence from initiation to document creation:
This diagram shows the flow of requests from the collector interface through the fallback client logic to the final document mapping.
Sources: lib/collectors/youtube.ts:15-53, lib/youtube/client.ts:400-435
Data Acquisition Methods
The collector handles different input types to trigger data retrieval.
| Configuration Field | Type | Description |
|---|---|---|
videoIds | string[] | Array of specific YouTube video IDs to process. |
query | string | Search query used to discover relevant videos. |
maxResults | number | The limit of comments to retrieve per session. |
Sources: lib/collectors/youtube.ts:16-22, lib/youtube/client.ts:37-62
Discovery Logic
When you provide a query instead of videoIds, the system initiates a discovery path. It uses tinyFishSearch to find relevant video URLs, extracts the IDs, and then fetches comments for those specific videos.
The discovery flow identifies video candidates before executing the standard comment extraction logic.
Sources: lib/youtube/client.ts:326-350
Data Transformation and Storage
The collector transforms raw YouTubeComment objects into the project's core NewDocument format. This process normalizes data across different platforms.
Mapping Schema
| YouTube Field | Document Field | Transformation Logic |
|---|---|---|
id | rawSnapshotRef | Prefixed with yt:. |
text | contentMd | Combined with Video Title and Author. |
author | authorRef | Stored as the author identifier. |
publishedAt | postedAt | Parsed into a valid ISO date. |
videoUrl + id | urlCanonical | Combines video URL with comment ID fragment. |
videoId, likeCount | metadata | Stored in the JSONB metadata field. |
Sources: lib/collectors/youtube.ts:30-51, drizzle/0000_m1_core.sql:43-58
Content Formatting
The collector generates a Markdown body for each comment. It includes the video title and author at the top of the content for context during analysis.
// lib/collectors/youtube.ts:31-36
const body = [
`Video: ${c.videoTitle}`,
`Author: ${c.author}`,
"",
c.text,
].join("\n");
Sources: lib/collectors/youtube.ts:31-36
Provider Fallback Logic
The fetchYouTubeCommentsWithMeta function manages provider failures. If a high-tier provider fails or returns no results, the client logs a warning and attempts the next path.
This diagram represents the decision tree for selecting a data provider based on environment configuration and runtime availability.
Sources: lib/youtube/client.ts:400-435, test/tinyfish-collectors.test.ts:98-150
Summary
The YouTube Collector provides a resilient mechanism for ingesting social proof and developer feedback from video comments. By utilizing a tiered provider strategy, the system maintains high data availability even when specific external services fail. The transformation logic ensures that YouTube data integrates seamlessly with the project's internal document and opportunity analysis pipelines.