Social Media Scraping
Relevant source files
The following files were used as context for generating this wiki page:
Social Media Scraping is a core capability of the Recoup API designed to provide real-time, fresh data on artist performance, fan engagement, and cultural trends across platforms like Instagram, TikTok, X (formerly Twitter), and YouTube. This system bypasses stale database values by utilizing a suite of specialized scrapers and deep research tools to fetch live metrics such as follower counts, song usage in short-form video content, and press mentions.
The system is integrated into the AI assistant's workflow, allowing it to proactively surface trends and optimize marketing strategies for artist teams. It leverages external services like Apify and Perplexity to perform deep web research and platform-specific data extraction.
Sources: lib/chat/const.ts:34-40, evals/social-scraping.eval.ts:11-20
Architecture and Components
The scraping architecture is modular, relying on a combination of platform-specific tools and general-purpose web research agents. The system utilizes a "Sandbox-First" approach where complex data analysis tasks are performed in a persistent environment.
Core Scraping Tools
The system employs several specialized tools to handle different types of data requests:
| Tool Category | Key Tools | Purpose |
|---|---|---|
| Platform Scrapers | scrape_instagram_profile, scrape_instagram_comments, search_twitter | Extraction of profile metrics, engagement data, and trend analysis from specific social networks. |
| Web Research | search_web, web_deep_research, get_apify_scraper | General web searching and execution of custom Apify actors for diverse data needs. |
| Deep Analysis | sonar-deep-research (Perplexity) | Complex research tasks requiring synthesis of multiple web sources and citations. |
Sources: evals/social-scraping.eval.ts:28-36, lib/mcp/tools/tests/registerWebDeepResearchTool.test.ts:43-52, lib/chat/const.ts:10-15
Data Flow for Scraping Requests
The following diagram illustrates how a user request for social data is processed through the scraping layer:
The system prioritizes fresh data from these scrapers over internal database values to ensure accuracy in fast-moving social environments.
Sources: lib/chat/const.ts:10-25, evals/social-scraping.eval.ts:48-60
Platform Specific Implementations
YouTube Data Extraction
YouTube data is fetched using authenticated API clients. The system retrieves channel statistics, branding settings, and upload information.
- Key Function:
fetchYouTubeChannelInfo - Data Points: Subscriber counts, video counts, view counts, and custom URLs.
- Authentication: Utilizes
accessTokenandrefreshTokenvia a dedicated OAuth client.
Sources: lib/youtube/fetchYouTubeChannelInfo.ts:4-30
Deep Web Research (Perplexity Integration)
For broad queries (e.g., finding press interviews or visibility moments), the system uses the web_deep_research tool. This tool interfaces with Perplexity's sonar-deep-research model to provide cited research findings.
Scraping Evaluation and Quality Assurance
To ensure the reliability of scraping operations, the project includes an evaluation suite (social-scraping.eval.ts). This suite tests the AI's ability to:
- Exhaust all options: Attempting multiple scraping tools before reporting failure.
- Avoid stale data: Explicitly checking that the AI does not return "0" from a database when live data is available.
- Specific Output: Ensuring the AI provides specific numbers (e.g., "112,545 followers") and lists of content (e.g., specific IG Reels causing a spike).
Sources: evals/social-scraping.eval.ts:10-25
Dependencies and Infrastructure
The scraping capabilities are supported by the following core dependencies:
apify-client: Interface for running social media extraction actors.googleapis: Used for YouTube channel and video data retrieval.braintrust: Used for running and scoring scraping evaluations.ai: The core AI SDK facilitating tool calls and response generation.
Sources: package.json:37-55
Conclusion
The Social Media Scraping system is a critical component for the "Sandbox-First" strategy of the Recoup API. By integrating platform-specific scrapers with deep research agents, the system provides artist teams with actionable, real-time intelligence that extends beyond static database records. This allows for proactive identification of marketing opportunities, such as viral spikes on Instagram Reels or emerging trends on X.
Sources: lib/chat/const.ts:10-18, evals/social-scraping.eval.ts:11-20