Data Labeling and Correction

Relevant source files

The following files were used as context for generating this wiki page:

• src/labeling/interactive_labeler.py
• src/labeling/semi_auto_labeler.py
• fix_labelling.py
• src/preprocessing/fix_labels.py
• src/preprocessing/improve_labels.py
• src/modeling/apply_hybrid_classifier.py
• src/analysis/analyze_labeling.py
• src/preprocessing/optimized_sentiment_labeler.py
• config/sentiment_config_v2_optimized.py
• src/labeling/retrain_sentiment.py

The Data Labeling and Correction system is a multi-stage process designed to assign accurate sentiment and categorical labels to raw comment data. Its primary goal is to reduce the number of 'unknown' labels, simplify complex label hierarchies, and improve the overall quality of the dataset for more effective model training and analysis. This is achieved through a combination of automated rule-based labeling, semi-automated suggestion tools, interactive manual labeling, and machine learning-based prediction for unclassified data.

The system employs a layered approach, starting with broad sentiment classification and progressively adding more granular labels related to criticism targets, root causes, and constructiveness. Various scripts work in concert to analyze, clean, label, and retrain models on the data, forming a comprehensive pipeline from raw text to a high-quality, labeled dataset.

Overall Labeling and Correction Workflow

The system follows a comprehensive workflow that combines automated, semi-automated, and manual processes to refine the dataset. It begins with initial rule-based labeling, identifies comments that could not be classified ('unknown'), provides tools for manual intervention, and uses machine learning to fill the remaining gaps.

The following diagram illustrates the high-level data flow through the various labeling and correction scripts.

Semi-Automated Labeling

This stage focuses on processing comments that were initially marked as 'unknown' by the primary labeling system. The SemiAutoLabeler class is used to suggest potential labels based on pattern matching, preparing the data for efficient manual review.

Sources: src/labeling/semi_auto_labeler.py

Architecture and Flow

The SemiAutoLabeler works by:

1. Loading a CSV file and isolating rows where sentiment_label is 'unknown'.
2. Building a set of predefined regular expression patterns for various categories.
3. Iterating through unknown comments and matching them against the patterns to suggest a label.
4. Analyzing the characteristics of unknown comments (e.g., word frequency, length).
5. Exporting the unknown comments, along with their suggested labels, to a new CSV file for manual correction.

This process is visualized in the following flowchart:

Key Components

• SemiAutoLabeler Class: The main class encapsulating the semi-automated labeling logic.
  Sources: src/labeling/semi_auto_labeler.py:8
• _build_patterns(): Defines the regex patterns used for suggesting labels. These patterns are grouped by category.
  Sources: src/labeling/semi_auto_labeler.py:15-48
• suggest_labels(): The core method that applies the patterns to the text and generates label suggestions.
  Sources: src/labeling/semi_auto_labeler.py:50-74
• export_for_manual_labeling(): Creates a new CSV file (*_manual_labeling.csv) containing the original comment, the suggested label, and an empty manual_label column to be filled in by a user.
  Sources: src/labeling/semi_auto_labeler.py:114-140

Interactive Manual Labeling

For comments that require human review, the InteractiveLabeler provides a command-line interface to quickly and efficiently assign the correct labels. It is designed to work with the output from the SemiAutoLabeler.

Sources: src/labeling/interactive_labeler.py

User Interaction Flow

The tool presents one comment at a time, displays any machine-generated suggestions, and provides a menu of possible labels. The user can then input a number corresponding to a label, accept the suggestion, or skip the comment. Progress is automatically saved.

This sequence diagram illustrates the interaction:

Key Components

• InteractiveLabeler Class: Manages the state of the labeling session, including the current index and any changes made.
  Sources: src/labeling/interactive_labeler.py:6
• LABELS Dictionary: A mapping from user input (e.g., '1', 's', 'a') to the actual label string or command.
  Sources: src/labeling/interactive_labeler.py:9-26
• Progress Persistence: The tool saves its state (current index and labeled changes) to a _progress.json file. This allows the user to stop and resume the labeling process without losing work.
  Sources: src/labeling/interactive_labeler.py:32-46
• apply_changes(): This method is called at the end of a session to write the manually assigned labels back to the CSV file.
  Sources: src/labeling/interactive_labeler.py:145-160

Automated Label Correction and Simplification

Several scripts are dedicated to programmatically cleaning up labels. This involves fixing 'unknown' values using keyword-based rules and simplifying overly granular label sets into more manageable, higher-level categories. This step is crucial for creating a consistent dataset for model training.

Sources: fix_labelling.py, src/preprocessing/fix_labels.py, src/preprocessing/improve_labels.py

Label Fixing and Simplification Logic

The scripts use predefined mappings to consolidate and correct labels.

• fix_labelling.py:
  • Uses a keyword-based function (classify_target) to assign a target_kritik to comments where it is 'unknown'.
  • Simplifies football_emotion labels into a smaller set (e.g., 'patriotic_sadness' -> 'disappointed').
  • Classifies constructiveness based on sentiment and keywords like 'saran' (constructive) or 'bodoh' (destructive).
• src/preprocessing/fix_labels.py & src/preprocessing/improve_labels.py:
  • These scripts define explicit mapping dictionaries to merge multiple specific labels into broader categories. For example, various negative emotions are mapped to a single 'negative' sentiment.

The following table shows an example of label simplification from src/preprocessing/improve_labels.py.

Original emotion_label	Simplified sentiment_simple
positive_support	positive
respectful_acknowledgment	positive
future_hope	positive
negative_criticism	negative
passionate_disappointment	negative
patriotic_sadness	negative
neutral_observation	neutral

Sources: src/preprocessing/improve_labels.py:14-30

Model Retraining on Corrected Labels

After fixing the labels, scripts like fix_labelling.py and src/labeling/retrain_sentiment.py proceed to train or retrain machine learning models on this cleaner data.

• fix_labelling.py trains a quick SVC model for each of the corrected target columns (core_sentiment, target_kritik, etc.).
• src/labeling/retrain_sentiment.py trains an ensemble VotingClassifier (Logistic Regression + Naive Bayes) on the labeled data to predict sentiments for the remaining 'unknown' entries.
• src/preprocessing/improve_labels.py uses SMOTE to handle class imbalance before training an ensemble model (SVC, RandomForestClassifier, MultinomialNB).

Sources: fix_labelling.py:72-111, src/labeling/retrain_sentiment.py:68-111, src/preprocessing/improve_labels.py:61-92

Hybrid Classification for Unknowns

After initial rule-based and manual labeling, a significant number of comments might still be marked 'unknown' in deeper layers of analysis (e.g., root_cause). The HybridClassifierApplicator is designed to address this by using pre-trained machine learning models to predict these missing labels, with the goal of reducing the 'unknown' rate to below 10%.

Sources: src/modeling/apply_hybrid_classifier.py

Architecture

1. Load Models: The system loads previously trained classifiers and their corresponding TfidfVectorizers for each label layer (e.g., layer3_cause, layer4_time).
2. Load Data: It reads the dataset that has already undergone initial labeling (e.g., v4_phrases.csv).
3. Predict with Confidence: For each layer, it identifies 'unknown' comments, transforms their text using the appropriate vectorizer, and predicts a label.
4. Confidence Filtering: A crucial step is calculating a confidence score for each prediction. Predictions below a set threshold (e.g., 40%) are rejected, and the label remains 'unknown'.
5. Apply Labels: High-confidence predictions are used to update the dataset, creating new _v5 columns.
6. Analyze & Save: The system calculates the reduction in 'unknown' labels and saves the final, enriched dataset.

The workflow is depicted below:

Confidence Calculation

For LinearSVC models, the confidence score is derived from the decision_function. The raw scores (distance to the hyperplane) are normalized using a sigmoid function to produce a value between 0 and 1. This allows the system to only accept predictions it is sufficiently "sure" about.

Sources: src/modeling/apply_hybrid_classifier.py:110-120

Labeling Quality Analysis

To ensure the effectiveness of the labeling process, the analyze_labeling.py script provides a comprehensive analysis of the final labeled dataset. It generates key metrics and offers recommendations for improvement.

Sources: src/analysis/analyze_labeling.py

Key Metrics Calculated

The analysis script computes several quality indicators:

Metric	Description
Label Distribution	The count and percentage of each sentiment label, highlighting the ratio of labeled vs. 'unknown' data.
Layer Distribution	The count and percentage of labels assigned by each layer of the sentiment framework.
Confidence Statistics	Mean, median, standard deviation, min, and max of the confidence scores for all labels.
High Confidence Rate	The percentage of comments labeled with a confidence score of 0.7 or higher.
Overall Quality Score	A weighted score calculated from Labeling Coverage (50%), Avg Confidence (30%), and High Conf Rate (20%).

Sources: src/analysis/analyze_labeling.py:15-68

Recommendations

Based on the metrics, the script provides actionable recommendations, such as:

• If the 'unknown' percentage is too high (>30%), it suggests adding more keywords to the lexicon.
• If the average confidence is low (<0.5), it recommends reviewing confidence thresholds and conflict detection logic.
• It flags if any confidence scores are above 1.0, indicating a potential bug in the calculation.

Sources: src/analysis/analyze_labeling.py:84-96

Conclusion

The data labeling and correction framework is a sophisticated, multi-faceted system that is fundamental to the project's success. By combining rule-based automation, ML-driven prediction, and human-in-the-loop verification, it transforms raw, unstructured comments into a high-quality, consistently labeled dataset. This rigorous process ensures that subsequent model training and sentiment analysis are built on a reliable and accurate foundation, leading to more meaningful and trustworthy insights.