Data Preprocessing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The project employs two distinct data preprocessing pipelines to prepare text data for sentiment analysis. The first is a Python-based batch processing pipeline used for training the SVM machine learning models. It focuses on cleaning text and converting it into numerical features using TF-IDF.
The second is a real-time preprocessing pipeline implemented in TypeScript within the Next.js dashboard. This pipeline is designed for live YouTube comments, featuring a sophisticated "Smart Comment Filter" to ensure only relevant, high-quality comments are analyzed by the live AI model. It also includes a lexicon-based fallback mechanism.
Batch Preprocessing Pipeline (Python)
This pipeline is orchestrated by the model training scripts and is designed to prepare a cleaned, pre-labeled dataset for training the SVM classifier.
Sources: src/pipeline/train_model.py, src/labeling/retrain_sentiment.py, src/evaluation/ablation_study.py
High-Level Flow
The following diagram illustrates the sequence of operations in the batch preprocessing and training pipeline.
Sources: src/pipeline/train_model.py:70-101
Data Loading and Initial Filtering
Before any text processing, the pipeline loads the dataset from a CSV file and performs initial filtering to create a clean set for training.
- Load Data: Reads a CSV file (e.g.,
comments_clean.csv). - Filter Nulls: Removes rows where the text or label columns are empty.
- Filter 'unknown': Excludes comments with an 'unknown' label to ensure the model is trained only on clearly classified data.
Sources: src/pipeline/train_model.py:59-68
Text Cleaning
A dedicated clean_text function is used to sanitize the comment text. While the function itself is defined in src/preprocessing/text_cleaner.py, its effectiveness is verified in the test suite.
The cleaning steps include the removal of:
- URLs
- User mentions (
@user) - Hashtags (
#hashtag)
# Example from tests/test_model.py
def test_text_cleaning():
"""Test text cleaning function."""
from src.preprocessing.text_cleaner import clean_text
dirty_text = "Timnas!!! 123 @user #hashtag http://link.com"
clean = clean_text(dirty_text)
assert "http" not in clean, "URL not removed"
assert "@" not in clean, "Mention not removed"
assert "#" not in clean, "Hashtag not removed"
Sources: tests/test_model.py:53-64
Feature Extraction (TF-IDF)
The core of the feature extraction process is converting the cleaned text into a numerical format that machine learning models can understand. The project uses the Term Frequency-Inverse Document Frequency (TF-IDF) method.
The FeatureExtractor class encapsulates a TfidfVectorizer from scikit-learn. The configuration has been optimized based on ablation studies.
| Parameter | Value | Description | Source |
|---|---|---|---|
max_features | 2000 - 5000 | Limits the vocabulary to the top N most frequent terms. | src/pipeline/train_model.py:40, src/evaluation/ablation_study.py:100 |
ngram_range | (1, 2) | Considers both single words (unigrams) and two-word pairs (bigrams). | src/labeling/retrain_sentiment.py:58, src/evaluation/ablation_study.py:46 |
sublinear_tf | True | Applies logarithmic scaling to term frequency to diminish returns for high-frequency words. | src/evaluation/ablation_study.py:50 |
min_df | 2 - 5 | Ignores terms that appear in fewer than N documents. | src/labeling/retrain_sentiment.py:59, src/evaluation/ablation_study.py:133 |
max_df | 0.6 - 0.95 | Ignores terms that appear in more than N% of documents (common words). | src/labeling/retrain_sentiment.py:60, src/evaluation/ablation_study.py:143 |
The ablation study (ablation_study.py) confirms that using TF-IDF with unigrams and bigrams provides a significant performance boost over a simple Count Vectorizer baseline.
Sources: src/pipeline/train_model.py:90-95, src/labeling/retrain_sentiment.py:52-68, src/evaluation/ablation_study.py
Live Preprocessing Pipeline (TypeScript)
The Next.js dashboard uses a separate, sophisticated real-time preprocessing pipeline for handling live YouTube comments. This pipeline is focused on relevance and quality control before hitting the primary AI analysis model.
Sources: dashboard-next/app/api/live-comments/route.ts
High-Level Flow
This diagram shows the flow for processing live comments, including the smart filter and fallback logic.
Smart Comment Filtering
The filterRelevantComments function is a key component that acts as a gatekeeper. It uses a scoring system to decide if a comment is relevant to the context of Indonesian football (timnas).
Filtering Logic
A relevance score is calculated for each comment based on the following criteria:
| Category | Condition | Score Change |
|---|---|---|
| Positive Score | Contains football keywords (e.g., 'timnas', 'pssi', 'pemain') | +3 |
| Contains opinion words (e.g., 'bagus', 'jelek', 'harusnya') | +2 | |
| Comment length > 30 characters | +1 | |
Contains a question mark ? | +1 | |
| Penalty Score | Contains spam keywords (e.g., 'subscribe', 'promo', '.com') | -5 |
| Comment is just emojis | -3 | |
| Comment length < 8 characters | -2 | |
| Contains 5+ repeated characters (e.g., 'haaaaaa') | -2 | |
| Is all caps (and > 10 chars) | -1 |
A comment is considered relevant only if its final relevanceScore is greater than 0. This system effectively removes spam, short/meaningless comments, and irrelevant content.
Sources: dashboard-next/app/api/live-comments/route.ts:80-149
Fallback Sentiment Analysis
If the primary AI model (/predict endpoint) is unavailable, the analyzeSentimentFallback function provides a basic sentiment analysis. This ensures the system remains functional.
- Method: It uses a predefined lexicon of positive and negative words, each with a weight (strong, medium, weak).
- Scoring: It calculates a positive and negative score based on the words found in the comment.
- Decision: The final sentiment ('positive', 'negative', or 'neutral') is determined by comparing the ratio of positive and negative scores against a threshold.
// Simplified lexicon structure
const sentimentLexicon = {
positive: {
strong: ['sangat bagus', 'luar biasa', ...],
medium: ['bagus', 'mantap', 'keren', ...],
weak: ['lumayan', 'cukup', 'ok', ...]
},
negative: {
strong: ['sangat jelek', 'sangat buruk', ...],
medium: ['jelek', 'buruk', 'kecewa', ...],
weak: ['kurang', 'tidak', 'bukan', ...]
}
};
Sources: dashboard-next/app/api/live-comments/route.ts:7-56
Conclusion
The project's dual-pipeline approach to data preprocessing is a key architectural strength. The robust, offline Batch Pipeline ensures that the historical SVM models are trained on high-quality, consistently formatted data, with feature engineering choices validated by ablation studies. In parallel, the agile Live Pipeline in the dashboard provides real-time relevance filtering and resilience, ensuring a high-quality user experience by focusing analysis on meaningful comments and providing a graceful fallback mechanism. This separation of concerns allows each pipeline to be optimized for its specific task: one for model accuracy and the other for real-time relevance and system robustness.