Model Training and Evaluation
Relevant source files
The following files were used as context for generating this wiki page:- src/modeling/train_model.py
- src/pipeline/train_model.py
- src/modeling/evaluation.py
- src/evaluation/benchmark_performance.py
- src/modeling/compare_models.py
- tests/test_model.py
- SVM_MODEL_DOCUMENTATION.md
- DUAL_MODEL_ARCHITECTURE.md
- dashboard-next/app/model/page.tsx
- src/labeling/retrain_sentiment.py
- config/sentiment_config.py
The model training and evaluation system is a core component of this project, responsible for creating, validating, and benchmarking the sentiment analysis models. The primary model used for static, historical analysis is a Support Vector Machine (SVM), which is trained and evaluated through a structured pipeline. The system also supports an ensemble approach and includes comprehensive scripts for performance benchmarking and comparison against other models like Lexicon-based and IndoBERT.
This dual-architecture approach uses the highly consistent and fast SVM model for analyzing the pre-labeled historical dataset displayed on the static dashboard, while a separate Gemini AI model is used for real-time analysis. This documentation focuses on the training and evaluation pipeline for the static SVM model.
Sources: DUAL_MODEL_ARCHITECTURE.md, src/pipeline/train_model.py
Training Pipeline Architecture
The model training process is orchestrated by a main pipeline script that automates data loading, feature extraction, model training, evaluation, and artifact storage. This ensures a reproducible and consistent workflow for generating new model versions.
The following diagram illustrates the end-to-end training pipeline.
This diagram shows the complete workflow managed by src/pipeline/train_model.py.
Sources: src/pipeline/train_model.py:1-166, src/modeling/train_model.py
Pipeline Execution
The training pipeline is executed via the command line, accepting several arguments to configure the process.
| Argument | Description | Default Value |
|---|---|---|
--input | Path to the preprocessed comments CSV file. | data/processed/comments_clean.csv |
--text-column | Column containing the processed text. | normalized_text |
--label-column | Column containing the sentiment labels. | sentiment_label |
--test-size | Proportion of the dataset to allocate for testing. | 0.2 |
--max-features | Maximum number of features for the TF-IDF vectorizer. | 5000 |
--tune-hyperparameters | A flag to enable hyperparameter tuning. | False |
--output-dir | Directory to save the trained model and other artifacts. | data/models |
Sources: src/pipeline/train_model.py:23-53
Model Training
The core of the training process involves data preparation, feature engineering, and fitting an SVM classifier. An alternative approach using an ensemble of Logistic Regression and Naive Bayes is also available for re-training purposes.
Sources: src/modeling/train_model.py, src/labeling/retrain_sentiment.py
Data Preparation
Before training, the raw sentiment labels are mapped into three broad categories: positive, negative, and neutral. This simplifies the classification task and improves model performance by grouping related sub-sentiments.
# src/modeling/train_model.py:53-68
def map_sentiment_category(label):
positive_categories = ['positive_support', 'respectful_acknowledgment', 'fans_supporters']
negative_categories = ['pssi_management', 'opponents', 'patriotic_sadness',
'negative_criticism', 'frustration_expression', 'passionate_disappointment']
neutral_categories = ['unknown', 'technical_performance', 'management_decisions',
'players', 'external_factors', 'short_term_analysis',
'tactical_issues', 'systemic_reform_calls', 'future_projection']
if label in positive_categories:
return 'positive'
elif label in negative_categories:
return 'negative'
else:
return 'neutral'
The data is then split into training and testing sets, using stratification to maintain the same proportion of sentiment labels in both sets.
Sources: src/modeling/train_model.py:53-76, src/pipeline/train_model.py:84-86
Feature Extraction and Model Fitting
The system uses a FeatureExtractor class, which is a wrapper around Scikit-learn's TfidfVectorizer, to convert text data into numerical features. The SVM model is then trained on these features.
The key components involved in this process are illustrated below.
This diagram shows the relationship between the main classes for feature extraction, model training, and evaluation.
Sources: src/pipeline/train_model.py:92-95, src/modeling/train_model.py:18-20, src/modeling/evaluation.py:16-20
Hyperparameter Tuning
To find the optimal model configuration, GridSearchCV is used to perform an exhaustive search over a specified parameter grid. The search is optimized for the f1_weighted score, which is a robust metric for imbalanced datasets.
The parameter grid for the SVM model is defined as:
# src/modeling/train_model.py:94-98
param_grid = {
'C': [1, 10],
'gamma': ['scale', 0.1],
'kernel': ['rbf']
}
The best parameters found during the search are then used to train the final model.
Sources: src/modeling/train_model.py:94-106, config/sentiment_config.py:319-325
Alternative Ensemble Model
For re-labeling unknown data, an alternative ensemble model is used. This model combines LogisticRegression and MultinomialNB using a VotingClassifier with soft voting. This approach is effective for predicting labels on data that was not manually annotated.
Sources: src/labeling/retrain_sentiment.py:83-92
Model Evaluation
After training, the model's performance is rigorously evaluated on the held-out test set. The ModelEvaluator class provides a comprehensive set of metrics.
Sources: src/modeling/evaluation.py
Evaluation Metrics
The primary metrics used to assess model performance are:
| Metric | Description |
|---|---|
| Accuracy | The proportion of correctly classified samples. |
| Precision | The ability of the classifier not to label a negative sample as positive. |
| Recall | The ability of the classifier to find all the positive samples. |
| F1-Score | The weighted average of Precision and Recall. |
| ROC-AUC | The area under the ROC curve; measures the ability to distinguish between classes. |
| Classification Report | A text report showing the main classification metrics on a per-class basis. |
| Confusion Matrix | A matrix showing the counts of true vs. predicted classifications. |
These metrics are calculated by the evaluate method in ModelEvaluator. The results are then saved to evaluation_results.json and can be displayed in a formatted report.
Sources: src/modeling/evaluation.py:39-82, dashboard-next/app/model/page.tsx:64-113
Testing and Validation
The project includes a test suite (tests/test_model.py) that validates the model against predefined quality gates. These tests ensure that the model meets minimum performance standards before deployment.
test_model_accuracy(): Asserts that the model's accuracy is at least 65%.test_confidence_threshold(): Asserts that the average prediction confidence is at least 70%.
Sources: tests/test_model.py:101-112
Performance Benchmarking
The src/evaluation/benchmark_performance.py script is dedicated to measuring the model's computational performance, which is crucial for understanding its behavior under load.
The benchmarking process is as follows:
This diagram outlines the steps in the performance benchmarking script.
Sources: src/evaluation/benchmark_performance.py
Key Benchmarks
- Inference Speed: Measures throughput (samples/sec) and latency (ms/sample) for various batch sizes. This helps determine the optimal batch size for production.
- Memory Usage: Uses
memory_profilerto find the peak memory consumption during prediction for large batches. - Scalability: Analyzes how processing time scales with the number of samples, confirming if the model's performance is linear.
The results are saved to performance_benchmark.csv and visualized in performance_benchmark.png.
Sources: src/evaluation/benchmark_performance.py:22-81
Model Comparison
The repository includes a script to compare the trained SVM model against a baseline Lexicon-based model and an expected high-performance IndoBERT model. This comparison provides context for the model's performance and guides future development.
The comparison is based on the following metrics:
| Model | Accuracy | F1-Score | Avg Confidence | Complexity |
|---|---|---|---|---|
| Lexicon-based | 65.0% | 63.0% | 53.0% | Low |
| SVM (Regularized) | 71.9% | 68.9% | 90.3% | Medium |
| IndoBERT | 85.0% | 84.0% | 92.0% | High |
The script recommends the SVM (Regularized) model for production due to its balance of accuracy and speed, while IndoBERT is recommended for research where accuracy is paramount.
Sources: src/modeling/compare_models.py:20-47, 131-149, data/models/model_comparison.json
Conclusion
The model training and evaluation framework is a robust and comprehensive system designed to produce reliable sentiment analysis models. It features a reproducible training pipeline, thorough evaluation using multiple metrics, and detailed performance benchmarking. The inclusion of model comparison provides valuable insights for strategic decisions, ensuring the right model is used for the right task—balancing accuracy, speed, and complexity. This systematic approach guarantees model quality and provides a solid foundation for the sentiment analysis features in the application.