Social Media Scraping Operations

Relevant source files

The following files were used as context for generating this wiki page:

• src/socials/scrapeAndPollSocials.ts
• src/socials/filterScrapableSocials.ts
• src/recoup/scrapeSocial.ts
• src/recoup/getScraperResults.ts
• src/tasks/proArtistSocialProfilesScrape.ts

Social Media Scraping Operations represent a core automated system within the Recoup ecosystem designed to keep artist profiles synchronized with their real-world social media metrics. The system functions as a series of background tasks that identify, filter, and extract data from various social platforms to provide up-to-date insights for the Recoup API and dashboard.

The scope of these operations includes scheduled discovery of professional-tier artists, batch processing of their associated social links, and a robust polling mechanism to handle the asynchronous nature of third-party scraping services like Apify.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:1-15, src/socials/scrapeAndPollSocials.ts:1-10

Task Orchestration and Scheduling

The primary entry point for scraping operations is a scheduled Trigger.dev task. This task orchestrates the entire lifecycle of a scraping run, from artist discovery to result aggregation.

Pro Artist Profile Scrape Task

This task is defined as a schedules.task that executes daily at midnight Eastern Time. It is specifically scoped to "Pro" tier artists to manage resource allocation and ensure priority data remains fresh.

Social Filtering and Validation

Not all social links associated with an artist are suitable for automated scraping. The system implements a filtering layer to separate scrapable platforms (like TikTok or Instagram) from platforms that may require different integration methods (like Spotify).

Logic and Data Structures

The filtering logic uses a Map<string, Social[]> to associate artist IDs with their respective social profiles. It returns an array of ScrapableSocial objects which contain the necessary metadata for the scraper API.

Field	Type	Description
artistId	string	The internal Recoup identifier for the artist.
socialId	string	The specific identifier for the social profile record.
username	string	The social media handle.
profile_url	string	The full URL to the social media profile.

Sources: src/socials/filterScrapableSocials.ts:10-30, src/tasks/proArtistSocialProfilesScrape.ts:47-52

Scraping and Polling Architecture

The scraping process is inherently asynchronous. The system utilizes the Recoup Social Scrape API to trigger jobs and a polling mechanism to retrieve the results once the third-party datasets are ready.

Batch Processing

To avoid overwhelming the API and to manage long-running tasks, socials are processed in batches (default size: 10). The system waits for an entire batch to complete or fail before initiating the next set of requests.

API Integrations

The operations rely on two primary internal API endpoints hosted via Recoup's API infrastructure.

Social Scrape Initiation

• Endpoint: POST https://api.recoupable.com/api/social/scrape
• Purpose: Triggers an Apify run for a specific social ID.
• Validation: Uses Zod to validate that the response contains a runId and datasetId.

Scraper Result Retrieval

• Endpoint: GET https://api.recoupable.com/api/apify/scraper
• Query Parameters: runId (string)
• Responses:
  • In-Progress: Returns a status and datasetId.
  • Completed: Returns status, datasetId, and an array of extracted data.

Sources: src/recoup/scrapeSocial.ts:13-17, src/recoup/getScraperResults.ts:19-23, src/recoup/getScraperResults.ts:46-60

Conclusion

Social Media Scraping Operations provide a scalable, batch-oriented framework for data extraction. By combining scheduled Trigger.dev tasks with asynchronous polling and tiered artist filtering, the system ensures that professional-grade social metrics are consistently updated while maintaining high reliability through batch isolation and detailed logging.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:70-85, src/socials/scrapeAndPollSocials.ts:80-95