Pro Artist Social Profiles Scrape

Relevant source files

The following files were used as context for generating this wiki page:

• src/tasks/proArtistSocialProfilesScrape.ts
• src/recoup/getProArtists.ts (Referenced in task implementation)
• src/artists/getBatchArtistSocials.ts (Referenced in task implementation)
• README.md
• AGENTS.md
• types/database.types.ts

The Pro Artist Social Profiles Scrape is a scheduled background task designed to automate the collection and updating of social media metadata for artists on the "pro" tier. This system ensures that the platform maintains up-to-date metrics and profile information, such as follower counts and bio data, which are critical for generating insights and personalized content like Pulses.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:5-13, README.md:43-43

Architecture and Workflow

The feature is implemented as a Trigger.dev v4 scheduled task. It follows a batch-processing pattern to handle multiple artists and their respective social media accounts across platforms like TikTok, YouTube, Instagram, and Spotify.

Execution Schedule

The task is configured to run daily at midnight Eastern Time. It has a maximum duration of one hour and is configured specifically to not retry upon failure to avoid redundant scraping attempts in a single window.

Configuration	Value
Schedule (Cron)	0 0 * * * (Daily at midnight)
Timezone	America/New_York
Max Duration	3600 seconds (1 hour)
Retry Strategy	maxAttempts: 1 (No retries)

Sources: src/tasks/proArtistSocialProfilesScrape.ts:7-16

Data Flow Logic

The process flows through four distinct stages: artist discovery, social mapping, filtering, and execution.

The workflow utilizes specialized utilities for each step, ensuring a clean separation of concerns between fetching database records and interacting with external scraping services.
Sources: src/tasks/proArtistSocialProfilesScrape.ts:18-72, AGENTS.md:38-46

Key Components and Functions

Data Retrieval Utilities

The system relies on internal Recoup API client functions and artist utilities to identify the target dataset.

• getProArtists(): Retrieves a list of account IDs that are classified as "pro" tier. If no artists are returned, the task fails immediately to prevent unnecessary resource allocation.
• getBatchArtistSocials(artistIds): Takes the list of IDs and returns a map of artists to their associated social media profiles.
• filterScrapableSocials(artistIds, map): Refines the list to include only those social platforms supported by the scraping engine.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:18-47

Execution Engine

The actual scraping and data persistence are handled by scrapeAndPollSocials.

• scrapeAndPollSocials(allSocials): Orchestrates the interaction with scraping agents. It initiates the scrape runs and polls for completion status.
• Status Monitoring: The task tracks "SUCCEEDED" vs "FAILED" statuses for each batch to provide a summary of the run's effectiveness.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:56-78

Data Structures

The system interacts with the socials table and related analytics schemas defined in the database.

Social Profile Schema (socials)

This table stores the metadata retrieved during the scraping process.

Field	Type	Description
id	string	Unique identifier for the social profile
username	string	Handle or username on the platform
followerCount	number	Total followers at last scrape
followingCount	number	Total accounts followed
avatar	string	URL to the profile picture
bio	string	Biography text
updated_at	string	Timestamp of the last successful update

Sources: types/database.types.ts:1850-1865

Social Platform Types

The scraper supports various platforms defined by the social_type enum.

Error Handling and Logging

The task includes several layers of observability:

1. Validation Check: Throws an error if getProArtists returns an empty set or fails.
2. Missing Socials Warning: Specifically logs warnings for artists who exist in the "pro" list but have no linked social accounts for visibility.
3. Completion Summary: Returns a structured object containing totalArtists, totalRuns, succeeded, and failed counts for reporting within the Trigger.dev dashboard.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:21-23, src/tasks/proArtistSocialProfilesScrape.ts:40-45, src/tasks/proArtistSocialProfilesScrape.ts:80-85

Summary

The Pro Artist Social Profiles Scrape acts as the primary data ingestion engine for artist social metrics. By running on a 24-hour cycle, it maintains the fresh data required for the platform's analytics and automated content creation pipelines. Its implementation as a Trigger.dev task provides native logging, scheduling, and error monitoring without the need for manual infrastructure management.

Sources: src/tasks/proArtistSocialProfilesScrape.ts:7-16, README.md:43-43