Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• dashboard-next/app/api/live-comments/route.ts
• src/modeling/apply_hybrid_classifier.py
• dashboard-next/app/api/layers/route.ts
• config/sentiment_config_v2_optimized.py
• dashboard-next/app/api/analytics/targets/route.ts
• dashboard-next/DASHBOARD_STRUCTURE.md
• README.md
• PERFORMANCE_OPTIMIZATION_REPORT.md

The project's data flow is characterized by a dual-path architecture, handling both static, pre-processed data and live, real-time data. The static path involves collecting, cleaning, and labeling a large dataset of YouTube comments, which is then served to the dashboard for historical analysis. The live path fetches comments directly from YouTube in real-time, processes them through an AI model for sentiment analysis and filtering, and displays them on dedicated live-monitoring pages.

This dual approach allows for a cost-effective and fast experience for historical data exploration (using a pre-trained SVM model on a static CSV file) while providing powerful, context-aware analysis for live data streams (using the Gemini AI model). Data is served to the Next.js frontend via a combination of Next.js API routes (for static data) and a FastAPI backend (for live AI inference).

Sources: README.md, CHANGELOG.md:10-21

High-Level Data Flow

The following diagram illustrates the two main data pipelines from source to the end-user's dashboard.

This diagram shows the two distinct data paths: the live flow (YouTube API -> Next.js Route -> FastAPI -> Live Pages) and the static flow (CSV -> Python Scripts -> CSV -> Next.js Routes -> Static Pages).

Sources: dashboard-next/app/api/live-comments/route.ts, src/modeling/apply_hybrid_classifier.py, dashboard-next/app/api/layers/route.ts, dashboard-next/DASHBOARD_STRUCTURE.md

Static Data Processing Pipeline

The static data, used for historical analysis across most dashboard pages, undergoes an offline processing and enrichment pipeline. This process aims to create a high-quality, labeled dataset.

The diagram below shows the steps involved in creating the final static dataset.

This flow details how the HybridClassifierApplicator class enhances an existing labeled dataset. It loads trained SVM models to predict labels for comments previously marked as 'unknown', but only accepts predictions with a confidence score above a set threshold (40%). This reduces the number of 'unknown' entries, enriching the dataset for analysis.

Sources: src/modeling/apply_hybrid_classifier.py:1-120

Hybrid Classification

The core of the static data enrichment is the HybridClassifierApplicator.

• Input: optimized_clean_comments_v4_phrases.csv
• Process:
  1. Loads pre-trained LinearSVC models and TfidfVectorizer for different label layers (layer3_cause, layer4_time, layer5_constructive).
  2. Identifies rows where labels are 'unknown'.
  3. Uses the appropriate model to predict a new label for these rows.
  4. Calculates a confidence score for each prediction using the model's decision_function.
  5. If the confidence is above the threshold (0.40), the new ML-predicted label is applied. Otherwise, it remains 'unknown'.
• Output: optimized_clean_comments_v5_hybrid.csv, which contains the newly enriched labels in columns like root_cause_v5.

Sources: src/modeling/apply_hybrid_classifier.py:22-29, 100-106, 233-237

Live Data Processing Pipeline

The live data flow is designed for real-time analysis and is orchestrated by the Next.js backend, which communicates with the YouTube API and a separate Python-based AI service.

Live Comment Analysis Sequence

The following sequence diagram shows the interactions between the user, the Next.js dashboard, the YouTube API, and the AI backend when fetching live comments.

This diagram shows that the Next.js API route acts as an orchestrator. It first fetches data from YouTube, then internally filters it for relevance, and finally sends each relevant comment to the FastAPI backend for AI-powered sentiment analysis before returning the complete, annotated data to the client.

Sources: dashboard-next/app/api/live-comments/route.ts:221-299

Smart Filtering and Fallback

The /api/live-comments endpoint includes sophisticated logic for data quality.

• Smart Filtering: If filter is not false, the filterRelevantComments function is applied. It uses a scoring system based on keywords (e.g., 'timnas', 'pssi', 'pemain'), opinion indicators, and spam detection to remove irrelevant comments. Only comments with a positive relevance score are kept.
• AI Analysis: If sentiment=true, the analyzeSentimentWithUnified function calls the FastAPI backend at http://localhost:8000/predict.
• Fallback Mechanism: If the FastAPI service is unavailable or fails, a local analyzeSentimentFallback function is used. This fallback function uses a lexicon-based approach to provide a basic sentiment score, ensuring the system remains operational.

Sources: dashboard-next/app/api/live-comments/route.ts:74-156, 192-218

Data Serving and API Layer

Data is exposed to the frontend via API endpoints. The project uses a hybrid API approach: Next.js API routes for serving static, aggregated data, and a FastAPI server for real-time ML inference.

API Endpoints

Endpoint	Backend	Data Source	Purpose
GET /api/live-comments	Next.js	YouTube API & FastAPI	Fetches, filters, and analyzes live YouTube comments.
POST /predict	FastAPI	Gemini/SVM Model	Performs real-time sentiment analysis on a given text.
GET /api/layers	Next.js	comments_fixed_labels.csv	Aggregates and serves statistics for different analysis layers (sentiment, target, emotion).
GET /api/analytics/targets	Next.js	comments_fixed_labels.csv	Provides detailed statistics grouped by the target_kritik column.

Sources: dashboard-next/app/api/live-comments/route.ts, README.md:158-165, dashboard-next/app/api/layers/route.ts, dashboard-next/app/api/analytics/targets/route.ts

Multi-Layer Caching Strategy

To ensure high performance, the application employs a multi-layer caching strategy, primarily for the static data endpoints.

This diagram shows the three layers of caching that prevent redundant data processing:

1. Server-Side Memory Cache: The API route caches the parsed CSV data in memory for 5 minutes.
2. HTTP Cache: The API response includes Cache-Control headers to allow browsers and CDNs to cache the result for 5 minutes.
3. Client-Side SWR Cache: The frontend uses SWR with a 5-minute deduplication interval, preventing repeated requests from the same user session.

Sources: PERFORMANCE_OPTIMIZATION_REPORT.md:21-29, 101-114

Data Structure: 5-Layer Sentiment Framework

The core of the analysis is a 5-layer framework defined in the configuration. This structure dictates how each comment is categorized, providing a multi-faceted view of public opinion.

Layer	Column Name	Description	Example Values
1. Core Sentiment	core_sentiment	The basic positive, negative, or neutral tone.	positive, negative, neutral
2. Target	target_kritik	Who or what is being criticized or praised.	pssi_management, coaching_staff, players
3. Root Cause	root_cause	The underlying reason for the sentiment.	tactical_strategic, technical_skill, mental_psychological
4. Time Perspective	time_perspective	The perceived timeline for a solution.	immediate, short_term, long_term
5. Constructiveness	constructiveness	Whether the comment offers a solution or is just a complaint.	constructive, destructive, hopeful

This framework is used to label the static dataset and provides the structure for the analytical visualizations on the dashboard.

Sources: config/sentiment_config_v2_optimized.py:11-255