Feature: Real-time Analysis Dashboard
Relevant source files
The following files were used as context for generating this wiki page:
The Real-time Analysis Dashboard is a sophisticated feature designed for live sentiment analysis of YouTube comments, specifically optimized for Indonesian football-related content. It operates on a dual-model architecture, leveraging the Google Gemini AI for real-time, context-aware analysis of raw, unprocessed comments. This contrasts with the main dashboard's static analysis, which uses a pre-trained Support Vector Machine (SVM) model on a cleaned, historical dataset.
This system provides two primary interfaces for real-time interaction: the Live Comments page for viewing and analyzing comments as they are fetched, and the Live Analysis page for monitoring batch processing of comments from a specific video stream. Key capabilities include a smart comment filter to isolate relevant content, an "aggressive" sentiment analyzer to minimize neutral results, and a robust fallback system for continuous operation.
Sources: README.md, dashboard-next/README.md, CHANGELOG.md:5-8
Architecture
The project employs a dual-model architecture to serve different analysis needs. The Real-time Analysis feature specifically utilizes the "Live Dashboard" path, which is powered by Google's Gemini AI.
Sources: README.md:12-33, dashboard-next/README.md:5-20
This diagram illustrates the two distinct data processing pipelines: the static analysis path using SVM and the real-time analysis path using Gemini AI.
Sources: README.md:12-33, dashboard-next/README.md:5-20
Model Comparison
| Feature | SVM Model (Static) | Gemini AI (Live) |
|---|---|---|
| Data Source | Pre-labeled & cleaned | Real-time & raw |
| Use Case | Historical analysis, research | Real-time monitoring, social listening |
| Context | Limited | Excellent |
| Reasoning | No | Yes |
| Cost | Free | API quota |
Sources: README.md:27-33
Core Components
The real-time feature is composed of two main frontend pages and a powerful backend API route that orchestrates data fetching and analysis.
Frontend Pages
Live Comments (/live-comments)
This page provides an interface for fetching and viewing YouTube comments in real-time. It includes interactive controls to tailor the analysis.
- Real-time Fetching: Fetches the latest comments from a specified YouTube video.
- AI Analysis Toggle: Enables or disables the "aggressive" sentiment analysis performed by the backend.
- Smart Filter Toggle: Enables or disables the AI-powered filter to show only relevant comments.
- Reasoning Display: Shows the rationale behind each sentiment classification provided by the AI model.
Sources: dashboard-next/app/live-comments/page.tsx, dashboard-next/README.md:33-40
Live Analysis (/realtime)
This page is designed for monitoring a stream of comments from a YouTube video. It allows for batch processing and visualization of sentiment trends as they emerge.
- Auto-Search: Finds the latest videos related to "timnas" (the national team).
- Batch Processing: Fetches and analyzes comments in batches.
- Stream Control: Provides
startandstopfunctionality for the analysis stream. - Interactive Visualizations: Displays real-time charts and metrics based on the incoming data.
Sources: dashboard-next/app/realtime/page.tsx, dashboard-next/README.md:42-49
Backend API: GET /api/live-comments
This is the central API endpoint that powers the real-time features. It fetches comments from YouTube, filters them for relevance, and performs sentiment analysis before returning the results to the client.
Sources: dashboard-next/app/api/live-comments/route.ts
API Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
videoId | string | lDtSjKb_8Jo | The ID of the YouTube video to fetch comments from. |
sentiment | boolean | false | If true, performs sentiment analysis on the comments. |
filter | boolean | true | If true (default), applies the smart comment filter. Set to false to disable. |
maxResults | integer | 200 | The maximum number of comments to fetch and process. |
Sources: dashboard-next/app/api/live-comments/route.ts:200-203
API Response Structure
When sentiment=true, the API returns a detailed object including the analyzed comments and summary statistics.
{
"comments": [
{
"id": "comment_id",
"text": "Comment text...",
"author": "Author Name",
"sentiment": "negative",
"confidence": 0.95,
"reasoning": "Enhanced fallback analysis (Unified Scorer unavailable)",
"model": "Unified-Fallback",
"relevanceScore": 5,
"filterReason": "Football context"
}
],
"total": 55,
"totalFetched": 200,
"videoId": "lDtSjKb_8Jo",
"isLive": true,
"filtered": true,
"sentimentAnalysis": {
"enabled": true,
"model": "Unified-Realtime",
"summary": { "negative": 40, "positive": 15 },
"avgConfidence": 0.88
}
}
Sources: dashboard-next/app/api/live-comments/route.ts:251-267, dashboard-next/app/api/live-comments/route.ts:272-284
Data Flow and Processing
The following diagram shows the sequence of operations when a user requests live comments with filtering and analysis enabled.
This sequence illustrates the end-to-end process from client request to the final structured response, including the crucial filtering and analysis steps.
Sources: dashboard-next/app/api/live-comments/route.ts
Smart Comment Filtering
A key innovation is the filterRelevantComments function, which uses a scoring system to sift through raw comments and identify those that are most relevant to the topic of Indonesian football.
Filtering Logic:
- Keyword Matching: Checks for a list of football-related keywords (e.g.,
timnas,pssi,shin tae-yong, player names). - Spam Detection: Identifies and penalizes comments containing spam phrases (e.g.,
subscribe,link bio,http). - Quality Checks: Penalizes comments that are too short, consist only of emojis, contain excessively repeated characters, or are in all caps.
- Opinion Indicators: Rewards comments that contain opinion words (e.g.,
bagus,jelek,harusnya). - Relevance Scoring: A score is calculated based on the above criteria. A comment is deemed relevant if its final score is greater than 0.
The final list of relevant comments is sorted by this relevance score.
Sources: dashboard-next/app/api/live-comments/route.ts:80-154
Unified Sentiment Analysis
The analyzeSentimentWithUnified function is responsible for sentiment classification. It primarily calls the backend Python API, which uses the Unified Scoring System.
- Model Type: It specifies
model_type: "realtime"in its request. This instructs the backend to use an "aggressive" scoring model variant, which is optimized for live analysis by reducing neutral classifications. - Fallback System: If the Python API (
localhost:8000) is unavailable or times out, the system gracefully degrades by calling theanalyzeSentimentFallbackfunction.
Sources: dashboard-next/app/api/live-comments/route.ts:157-193, UNIFIED_SCORING_SYSTEM.md:12-15, 21-23
Fallback Mechanism: analyzeSentimentFallback
This function provides a robust, self-contained sentiment analysis capability within the Next.js API route itself.
- Lexicon-Based Scoring: It uses a predefined lexicon of positive and negative words, each categorized by strength (strong, medium, weak).
- Weighted Calculation: Words are assigned weights (strong: 3, medium: 2, weak: 1) to calculate positive and negative scores for a given text.
- Unified Thresholds: A comment is classified based on the ratio of positive and negative scores. A sentiment is assigned if its score ratio exceeds 0.35, otherwise it is classified as neutral. This ensures a degree of consistency even during fallback.