Feature: Social Sentiment Analysis
Relevant source files
The following files were used as context for generating this wiki page:The Social Sentiment Analysis feature is a core component of the OnChain Sage Data Processing Layer. Its primary purpose is to ingest and analyze real-time social media data from X (formerly Twitter) to gauge market sentiment surrounding various crypto assets. This analysis is a critical input for the Signal Aggregator, which combines sentiment scores with on-chain market data to generate actionable trading signals.
The system is designed to be modular, supporting multiple sentiment analysis backends, including rule-based methods (VADER, keyword matching) and machine learning models (TensorFlow/Transformers). The feature includes data ingestion, text preprocessing, sentiment scoring, and a composite analysis that factors in social engagement metrics like likes and retweets.
Sources: ARCHITECTURE.md, apps/data-processing/processors/sentiment.py, apps/data-processing/processors/sentiment_analysis.py
Architecture and Data Flow
The sentiment analysis process follows a pipeline structure, starting from data ingestion and culminating in a normalized sentiment score. This score is then consumed by downstream services like the SignalAggregator.
The diagram below illustrates the high-level data flow for social sentiment analysis.
Data Ingestion
Social media data is collected from X (Twitter) using two primary methods:
- Official X API: The
sentiment_analysis.pyscript uses the official X API v2 recent search endpoint. It authenticates using a Bearer Token and is configured to handle rate limiting and retries. - Nitter Scraper: The
tf_sentiment.pyscript utilizes theNitterlibrary to scrape tweets without requiring API keys, providing an alternative data source.
The collected data includes the tweet text, user information, and engagement statistics (likes, retweets, comments), which are crucial for the composite analysis.
Sources: apps/data-processing/processors/sentiment_analysis.py:16-24, 252-282, apps/data-processing/models/sentiment/tf_sentiment.py:18-36
Sentiment Analysis Models
The system employs a modular design for sentiment analysis, allowing different backends to be used based on configuration. The core logic is defined by the SentimentAnalyzer abstract base class.
Sources: apps/data-processing/processors/sentiment.py:9-19
The following diagram shows the relationship between the different analyzer classes.
VaderSentimentAnalyzer
This analyzer uses the VADER (Valence Aware Dictionary and sEntiment Reasoner) library. It calculates polarity scores (positive, negative, neutral, compound) and normalizes the compound score to a 0-1 range.
Sources: apps/data-processing/processors/sentiment.py:21-48
TensorFlowSentimentAnalyzer
This implementation uses a pre-trained model from the Hugging Face library, specifically "distilbert-base-uncased-finetuned-sst-2-english". It leverages the pipeline API for sentiment analysis. The result label ('POSITIVE' or 'NEGATIVE') and score are combined into a final score between 0 and 1.
Sources: apps/data-processing/processors/sentiment.py:50-75
Keyword-Based Analyzers
Two keyword-based analyzers are available:
KeywordSentimentAnalyzer: Uses weighted dictionaries of positive and negative keywords. The final score is the ratio of the sum of positive keyword weights to the total sum of all matched keyword weights.CryptoSentimentAnalyzer: A more specialized version found insentiment_analysis.py. It uses unweighted sets of crypto-specific positive and negative keywords. The score is calculated as(positive_count - negative_count) / total_count.
Sources: apps/data-processing/processors/sentiment.py:77-124, apps/data-processing/processors/sentiment_analysis.py:30-143
An example of the crypto-specific keywords used:
| Type | Keywords |
|---|---|
| Positive | bullish, moon, hodl, gain, profit, breakthrough, adoption, utility |
| Negative | bearish, crash, dump, sell, scam, fraud, hack, sec, lawsuit |
Sources: apps/data-processing/processors/sentiment_analysis.py:34-85
Composite and Aggregated Analysis
CompositeSentimentAnalyzer
This class enhances the text-based sentiment analysis by incorporating social engagement metrics. It combines a base sentiment score from a selected backend (Vader, TensorFlow, etc.) with normalized scores from a tweet's likes, retweets, and comments.
The final score is a weighted average, as shown in the diagram below.
The weights for this calculation are configurable.
Sources: apps/data-processing/processors/sentiment.py:155-182, apps/data-processing/config/pipeline.yaml:20-25
Signal Aggregator Integration
The sentiment score is a key input for the SignalAggregator. This component calculates an aggregate score by combining the sentiment score with normalized market metrics (liquidity, volume, price action). The sentiment score is weighted according to the aggregator's configuration.
Sources: apps/data-processing/processors/signal_aggregator.py:168-189
TensorFlow LSTM Model Training
The file tf_sentiment.py outlines a complete pipeline for training a custom Bidirectional LSTM model for sentiment analysis. This provides a more tailored alternative to the generic pre-trained models.
The training process is as follows:
- Data Labeling: Since the scraped data is unlabeled, a pre-trained model (
distilbert-base-uncased-finetuned-sst-2-english) is used to assign initial "POSITIVE" or "NEGATIVE" labels. - Embeddings: The model uses pre-trained GloVe embeddings (
glove.6B.100d.txt) which are loaded into a non-trainableEmbeddinglayer. - Model Architecture: The model consists of an
Embeddinglayer, aBidirectional(LSTM(64))layer, and a finalDenselayer with a sigmoid activation for binary classification.
Sources: apps/data-processing/models/sentiment/tf_sentiment.py
Configuration
Sentiment analysis behavior is controlled via pipeline.yaml.
| Parameter | Type | Description | Default/Example |
|---|---|---|---|
sentiment.backend | string | The analyzer to use. Options: vader, tensorflow, keyword. | vader |
sentiment.weights.text | float | Weight for the text sentiment score. | 0.4 |
sentiment.weights.likes | float | Weight for the normalized likes count. | 0.2 |
sentiment.weights.retweets | float | Weight for the normalized retweets count. | 0.2 |
sentiment.weights.comments | float | Weight for the normalized comments count. | 0.2 |
sentiment.thresholds | object | Thresholds for categorizing the composite score. | strong_positive: 0.6 |
Sources: apps/data-processing/config/pipeline.yaml:18-29, apps/data-processing/processors/sentiment.py:133-136
API Endpoints
A Flask application in sentiment_analysis.py exposes endpoints for interacting with the sentiment analysis feature.
| Method | Endpoint | Description |
|---|---|---|
POST | /api/analyze/tweet | Analyzes the sentiment of a single tweet text provided in the JSON payload. |
POST | /api/ingest | Triggers a new ingestion and analysis of tweets from the X API. Can accept custom query parameters. |
POST | /api/analyze/file | Analyzes tweets from previously saved local .pkl files. |
Sources: apps/data-processing/processors/sentiment_analysis.py:355-416