External API Clients
Relevant source files
The following files were used as context for generating this wiki page:
The External API Clients are a core component of the Data Processing Layer in the OnChain Sage architecture. Their primary responsibility is to ingest real-time data from various external sources, including social media platforms like X (formerly Twitter) and on-chain data providers such as Raydium and Dex Screener. This data serves as the foundation for sentiment analysis, market metric extraction, and ultimately, the generation of trading signals.
These clients are designed to be modular and resilient, featuring a common base structure, configurable settings, and built-in mechanisms for rate limiting and retries. They abstract the complexities of interacting with different external APIs, providing a standardized way for the data processing pipeline to fetch necessary information.
Sources: ARCHITECTURE.md, apps/data-processing/README.md
Client Architecture
The client architecture is built upon an abstract base class, BaseClient, which defines a common interface and shared functionality for all specific client implementations. This promotes consistency and simplifies the creation of new clients for additional data sources.
Sources: apps/data-processing/clients/base.py
BaseClient
The BaseClient is an abstract class that all other clients inherit from. It establishes a contract for making HTTP requests and validating responses.
The following diagram shows the class hierarchy for the API clients.
Key Components:
__init__(self, config): The constructor takes a configuration dictionary, which typically includes the API endpoint, credentials, and retry/rate-limiting parameters._make_request(...): A protected method that handles the logic for making HTTP requests using therequestslibrary. It incorporates a retry mechanism usingtenacityto handle transient network errors or API failures.validate_response(self, response): An abstract method that must be implemented by each subclass. Its purpose is to ensure that the data received from the external API conforms to the expected format before it is processed further.
Sources: apps/data-processing/clients/base.py:7-31
Request and Retry Flow
The _make_request method in BaseClient is decorated with tenacity.retry, providing built-in resilience.
The diagram below illustrates the sequence of making a request through a client.
This flow ensures that temporary issues with an external API do not immediately cause the data ingestion process to fail. The retry logic is configured within the BaseClient.
Sources: apps/data-processing/clients/base.py:18-23
Client Implementations
The repository contains several concrete implementations of BaseClient for different data sources.
RaydiumClient
This client interacts with the Raydium API to fetch liquidity pool data. It includes a specific rate-limiting implementation to avoid exceeding the API's request limits.
Sources: apps/data-processing/clients/raydium_client.py
Key Methods:
| Method | Description |
|---|---|
get_pools() | Fetches all available liquidity pools. |
get_pool_by_id(id) | Fetches a specific liquidity pool by its ID. |
_rate_limit() | Pauses execution to adhere to the configured rate limit. |
validate_response(response) | Checks for success: true and a data list in the response. |
Sources: apps/data-processing/clients/raydium_client.py:38-106
The _rate_limit method ensures that the client does not send requests faster than the limit defined in the configuration.
# apps/data-processing/clients/raydium_client.py:48-54
def _rate_limit(self):
"""Implement rate limiting."""
current_time = time.time()
elapsed = current_time - self.last_request_time
if elapsed < 1.0 / self.rate_limit:
time.sleep(1.0 / self.rate_limit - elapsed)
self.last_request_time = time.time()
Sources: apps/data-processing/clients/raydium_client.py:48-54
DexScreenerClient
This client is responsible for fetching token pair information from the Dex Screener API.
Sources: apps/data-processing/clients/dex_screener_client.py
Key Methods:
| Method | Description |
|---|---|
get_pairs_by_address(addresses) | Fetches pair data for a given list of token addresses. |
validate_response(response) | Validates that the response contains a pairs field which is a list. |
The get_pairs_by_address method constructs a URL with multiple token addresses and retrieves market data such as price, volume, and liquidity for each.
Sources: apps/data-processing/clients/dex_screener_client.py:28-48
XClient
The XClient is used to search for recent tweets on X (formerly Twitter) based on a specified query. This is crucial for the social sentiment analysis pipeline.
Sources: apps/data-processing/clients/x_client.py
Key Methods:
| Method | Description |
|---|---|
search_tweets(query, max_results) | Searches for recent tweets matching the query. |
validate_response(response) | Checks if the response contains a data field. |
The client authenticates using a Bearer Token and queries the recent search endpoint. It processes the response to extract relevant tweet information like text, creation time, and public metrics (likes, retweets).
Sources: apps/data-processing/clients/x_client.py:28-54
Configuration
The configuration for all API clients is managed in a central YAML file, pipeline.yaml. This allows for easy modification of endpoints, credentials, and behavior without changing the code. Environment variables are used to securely inject sensitive information like API tokens.
Sources: apps/data-processing/config/pipeline.yaml
Below is a snippet of the configuration structure:
# apps/data-processing/config/pipeline.yaml
api:
x:
bearer_token: "${X_BEARER_TOKEN}"
endpoint: "https://api.x.com/2/tweets/search/recent"
query: '("crypto news" OR "crypto" OR tokens) lang:en -is:retweet'
max_results: 100
raydium:
endpoint: "https://api.raydium.io/v2"
rate_limit: 10 # requests per second
retry_attempts: 3
dex_screener:
endpoint: "https://api.dexscreener.com/latest"
rate_limit: 5 # requests per second
retry_attempts: 3
Sources: apps/data-processing/config/pipeline.yaml:1-17
The following table summarizes the key configuration parameters for each client:
| Client | Parameter | Description |
|---|---|---|
| x | bearer_token | The bearer token for authenticating with the X API. |
endpoint | The base URL for the X API's recent search endpoint. | |
query | The default search query to find crypto-related tweets. | |
max_results | The maximum number of tweets to fetch per request. | |
| raydium | endpoint | The base URL for the Raydium v2 API. |
rate_limit | The number of requests allowed per second. | |
retry_attempts | The number of times to retry a failed request. | |
| dex_screener | endpoint | The base URL for the Dex Screener API. |
rate_limit | The number of requests allowed per second. | |
retry_attempts | The number of times to retry a failed request. |
Sources: apps/data-processing/config/pipeline.yaml:1-17
Summary
The external API clients form the data ingestion backbone of the OnChain Sage platform. By leveraging a common BaseClient and centralized configuration, the system can reliably and consistently fetch data from diverse sources like Raydium, Dex Screener, and X. This modular design, complete with error handling and rate limiting, ensures that the upstream data processing and signal generation services have access to the timely and accurate data they need to function effectively.