Data Collection and Scraping

Relevant source files

The following files were used as context for generating this wiki page:

• src/scraper/comment_scraper.py
• src/scraper/youtube_search.py
• src/pipeline/data_collection.py
• dashboard-next/app/api/live-comments/route.ts
• dashboard-next/app/api/analytics/targets/route.ts
• create_correct_sentiment_data.py

The Data Collection and Scraping module is responsible for gathering video and comment data from YouTube. This forms the foundational dataset for all subsequent sentiment analysis tasks. The system is composed of two primary mechanisms: a Python-based batch pipeline for large-scale historical data collection and a Next.js-based real-time scraper for live dashboard features.

The batch pipeline systematically searches for relevant videos, filters them based on specific criteria, and scrapes a target number of comments. The live scraper, on the other hand, provides on-demand comment fetching for specific videos, enabling features like the "Live Comments" page on the dashboard. Both systems interact with the YouTube Data API v3 and include logic for error handling, rate limiting, and data structuring.

System Architecture

The data collection architecture is split into a batch processing pipeline and a real-time fetching service.

This diagram illustrates the two distinct data collection flows: the Python-based batch pipeline for creating the main dataset and the Next.js API for live data on the dashboard.

Sources: src/pipeline/data_collection.py, dashboard-next/app/api/live-comments/route.ts

Batch Data Collection Pipeline

The batch pipeline is orchestrated by src/pipeline/data_collection.py and utilizes the YouTubeSearcher and CommentScraper classes to build the primary dataset.

The flowchart above shows the step-by-step process executed by data_collection.py to search, filter, and scrape data.

Sources: src/pipeline/data_collection.py:101-163

Video Search (YouTubeSearcher)

The YouTubeSearcher class in src/scraper/youtube_search.py is responsible for finding relevant videos.

• Functionality: It iterates through a predefined list of SEARCH_QUERIES, sends requests to the YouTube API, and retrieves a list of video search results.
• Details Fetching: After finding videos, it uses get_video_details to fetch comprehensive metadata, including statistics and content details, for a list of video IDs.
• Filtering: A crucial step is filter_videos_by_criteria, which filters videos based on view count, comment count, duration, and language to ensure data quality.

# Example of filtering criteria from youtube_search.py
def _meets_criteria(self, snippet: Dict, statistics: Dict, content_details: Dict) -> bool:
    # View count filter (minimum 10,000 views)
    view_count = int(statistics.get('viewCount', 0))
    if view_count < 10000:
        return False
        
    # Comment count filter (minimum 100 comments)
    comment_count = int(statistics.get('commentCount', 0))
    if comment_count < 100:
        return False
        
    # Duration filter (4-20 minutes)
    duration = content_details.get('duration', '')
    if not self._is_valid_duration(duration):
        return False

Sources: src/scraper/youtube_search.py:127-154

Comment Scraping (CommentScraper)

The CommentScraper class in src/scraper/comment_scraper.py handles the collection of comments from the filtered videos.

• Core Method: get_video_comments fetches comments for a single video ID, handling pagination and rate limiting.
• Orchestration: scrape_videos_comments iterates through a list of video IDs, calling get_video_comments for each until a TARGET_COMMENTS goal is met.
• Features: It includes robust features like quota tracking, automatic retries with exponential backoff, periodic backups, and data extraction logic to structure the comment data.

Comment Data Structure

The scraper extracts specific fields for each comment, which are then saved to a CSV.

Field	Type	Description
comment_id	string	The unique ID of the comment.
video_id	string	The ID of the video the comment belongs to.
text	string	The displayed text of the comment.
author	string	The display name of the comment author.
like_count	integer	Number of likes on the comment.
reply_count	integer	Number of replies to the comment.
published_at	string	ISO 8601 timestamp of when the comment was published.
is_reply	boolean	Flag indicating if the comment is a reply.

Sources: src/scraper/comment_scraper.py:168-191

Pipeline Orchestration (data_collection.py)

This script serves as the entry point for the batch data collection process. It coordinates the YouTubeSearcher and CommentScraper to perform the end-to-end pipeline:

1. Searches for videos using all predefined queries.
2. Gathers unique video IDs and fetches their detailed metadata.
3. Filters the videos and saves the metadata to videos.csv.
4. Selects a final list of videos to scrape based on comment targets.
5. Iteratively scrapes comments from these videos until target_comments is reached.
6. Saves all collected comments to comments.csv.
7. Generates a collection_summary.json with metrics about the run.

Sources: src/pipeline/data_collection.py:101-163

Real-time Comment Scraping

For live dashboard features, the project uses a Next.js API route at dashboard-next/app/api/live-comments/route.ts to fetch comments on-demand.

This sequence diagram shows the flow for a live comment request from the user's browser to the Next.js backend and the YouTube API.

Sources: dashboard-next/app/api/live-comments/route.ts

API Endpoint: GET /api/live-comments

This endpoint fetches the most recent comments for a given YouTube video.

Parameters:

Parameter	Type	Default	Description
videoId	string	lDtSjKb_8Jo	The ID of the YouTube video to scrape comments from.
sentiment	boolean	false	If true, performs sentiment analysis on the comments.
filter	boolean	true	If true (default), applies the smart relevance filter.
maxResults	integer	200	The maximum number of comments to fetch.

Sources: dashboard-next/app/api/live-comments/route.ts:197-201

Relevance Filtering

The live scraper includes a sophisticated filterRelevantComments function that scores and filters comments based on a set of rules to ensure only relevant content is displayed.

Filtering Logic:

• Context Check: A relevance score is increased if the comment contains football-related keywords (timnas, pssi, pemain, gol, etc.).
• Opinion Check: The score is increased if opinion words (bagus, jelek, suka, kecewa) are present.
• Quality Penalties: The score is decreased for spam keywords, being too short, containing only emojis, or having excessive repeated characters.
• Final Decision: Only comments with a positive final score are kept.

Sources: dashboard-next/app/api/live-comments/route.ts:68-144

Final Data Output

The collected data is processed and stored in CSV files, which are then used by the dashboard and modeling scripts. The final structure includes the raw text and various labels derived from analysis. The file comments_fixed_labels.csv is read by API routes like /api/analytics/targets to generate visualizations.

Column	Description	Source File
clean_text	The processed comment text.	dashboard-next/app/api/analytics/targets/route.ts:25
core_sentiment	The primary sentiment label (positive, negative, neutral).	dashboard-next/app/api/analytics/targets/route.ts:35
target_kritik	The main target of the criticism in the comment.	dashboard-next/app/api/analytics/targets/route.ts:33

This table shows key columns from the final processed data file as used in the analytics API.

Sources: dashboard-next/app/api/analytics/targets/route.ts, create_correct_sentiment_data.py

Summary

The project employs a dual-pronged approach to data collection. A robust, configurable Python pipeline handles the heavy lifting of creating a large, high-quality historical dataset. This is complemented by a nimble, real-time Next.js service that powers the interactive features of the dashboard. This separation of concerns allows for both deep, offline analysis and responsive, live user experiences.