Feature: On-Chain Data Analysis
Relevant source files
The following files were used as context for generating this wiki page:- apps/data-processing/clients/dex_screener_client.py
- apps/data-processing/clients/raydium_client.py
- apps/data-processing/processors/raydium/raydium_processor.py
- ARCHITECTURE.md
- apps/data-processing/config/pipeline.yaml
- apps/data-processing/processors/forum_pipeline.py
- README.md
- apps/data-processing/README.md
Introduction
On-Chain Data Analysis is a core feature of the OnChain Sage platform, responsible for ingesting, processing, and analyzing market data directly from blockchain sources. It works in tandem with social sentiment analysis to generate comprehensive and reliable trading signals. This system primarily interacts with decentralized exchange (DEX) APIs like Raydium and Dex Screener to gather real-time metrics such as liquidity, volume, and price action.
The processed on-chain data is a critical input for the signal generation and aggregation algorithms. By combining on-chain metrics with social trends, OnChain Sage aims to provide traders with actionable insights into high-potential crypto tickers. The entire process is managed within the modular Data Processing Layer, ensuring scalability and maintainability.
Sources: ARCHITECTURE.md:4-8, README.md:23-24, apps/data-processing/README.md:3-5
Architecture and Data Flow
The on-chain analysis functionality resides within the Data Processing Layer, a Python-based environment designed for data ingestion, analysis, and aggregation. This layer is architecturally distinct, promoting loose coupling with other system components like the frontend and backend.
The general data flow for on-chain analysis is as follows:
- Data Ingestion: Client modules connect to external APIs (Raydium, Dex Screener) to fetch raw on-chain data.
- Data Processing: Processor modules validate, normalize, and extract key metrics from the raw data.
- Signal Aggregation: The processed on-chain metrics are combined with other data sources, such as social sentiment scores.
- Storage & Delivery: The final, aggregated data and generated signals are persisted in a PostgreSQL database and cached in Redis. They are then delivered to the backend for consumption by the user-facing dashboard.
Sources: ARCHITECTURE.md:12-16, ARCHITECTURE.md:55-62, apps/data-processing/README.md:3-16
Data Flow Diagram
The following diagram illustrates the flow of on-chain data from external sources through the system to the end-user.
Sources: ARCHITECTURE.md:12-16, ARCHITECTURE.md:80-97
Data Sources and Configuration
The system is configured to pull data from multiple on-chain sources. The primary sources are defined in the pipeline.yaml configuration file.
Sources: apps/data-processing/config/pipeline.yaml:9-18
API Client Configuration
Each data source has specific configuration parameters, including its API endpoint, rate limits, and retry policies. This ensures robust and respectful interaction with third-party services.
| Parameter | Raydium | Dex Screener | Description |
|---|---|---|---|
endpoint | https://api.raydium.io/v2 | https://api.dexscreener.com/latest | The base URL for the API. |
rate_limit | 10 req/s | 5 req/s | The maximum number of requests per second. |
retry_attempts | 3 | 3 | The number of times to retry a failed request. |
Sources: apps/data-processing/config/pipeline.yaml:9-18
The client modules, such as raydium_client.py and dex_screener_client.py, are responsible for implementing the logic to connect to these endpoints and respect the configured rate limits.
Sources: apps/data-processing/README.md:8-10
Data Processing Pipeline
Once data is ingested by the clients, it is passed through a processing pipeline. While a specific raydium_processor.py exists, the forum_pipeline.py provides a clear example of the generic pipeline structure.
The key steps in a typical processing pipeline are:
- Parse/Validate: Ingest raw data and ensure it conforms to the expected format.
- Enrich: Fetch additional data required for analysis, such as current asset prices.
- Compute: Calculate derived metrics, like profitability or other technical indicators.
- Aggregate: Combine various data points (e.g., on-chain metrics and sentiment scores) into a unified structure.
- Dispatch: Send the processed data to other systems, such as the signal storage or a post-management system.
Sources: apps/data-processing/processors/forum_pipeline.py:7-113
Example Pipeline Flow
This flowchart, based on the forum_pipeline.py logic, shows the sequence of operations for processing an incoming piece of data.
Signal Generation
The processed on-chain data is a crucial component in the final signal generation. The pipeline.yaml file defines how different data types are weighted to calculate a final confidence score for a trading signal.
Signal Weighting
The signal configuration section specifies the contribution of each component to the signal's confidence score.
| Component | Weight | Description |
|---|---|---|
sentiment | 0.3 | The score derived from social media analysis. |
market_metrics | 0.4 | The score derived from on-chain data (volume, liquidity, etc.). |
forum_profitability | 0.3 | The score derived from analyzing trading calls on forums. |
These weights are used by the Signal Aggregator to produce a single, actionable confidence score. The system then categorizes signals as "high," "medium," or "low" confidence based on predefined thresholds.
Sources: apps/data-processing/config/pipeline.yaml:37-41
Conclusion
The On-Chain Data Analysis feature is fundamental to OnChain Sage's mission of delivering data-driven trading insights. By systematically ingesting and processing data from reliable sources like Raydium and Dex Screener, the platform builds a quantitative foundation for its trading signals. The modular architecture of the Data Processing Layer, combined with a configurable pipeline and weighting system, creates a robust and extensible framework for analyzing market dynamics and empowering traders.
Sources: ARCHITECTURE.md:4-8, README.md:19-21