Polling & Async State Hooks

Polling and asynchronous state hooks are critical components in the Recoup ecosystem, designed to manage long-running background processes. Since many operations—such as social media scraping, AI video generation, and sandbox execution—occur asynchronously, the system implements robust polling mechanisms to track state transitions and retrieve results once tasks reach a terminal state.

These hooks are primarily integrated with Trigger.dev v4, utilizing native polling capabilities (runs.poll()) and custom retry logic to ensure reliable data retrieval across distributed environments.

Sources: src/tasks/pollContentRunTask.ts:11-15, src/polling/pollScraperResults.ts:20-25, CLAUDE.md:28-30

Polling Architectures

The project utilizes several distinct polling patterns depending on the resource being monitored. These include parallel polling for batch jobs, sequential batch polling to avoid rate limits, and state-based polling for infrastructure components like sandboxes.

Parallel Task Polling

For content creation, the system triggers multiple sub-tasks and polls them concurrently. This is handled by pollContentRuns, which maps multiple Run IDs to runs.poll() promises.

The diagram above illustrates how the system waits for multiple content generation runs to finish before aggregating their success or failure status.
Sources: src/content/pollContentRuns.ts, src/tasks/pollContentRunTask.ts:25-31

Scraper Result Polling

The social scraper polling logic uses a while loop with a manual wait interval. It maintains a pendingRuns map and removes entries as they reach SUCCEEDED or FAILED statuses.

Sources: src/polling/pollScraperResults.ts:21-72

Infrastructure State Monitoring

Sandbox Status Polling

Sandboxes are isolated microVMs that must be monitored until they transition to a stopped state. The pollSandboxUntilStopped utility performs HTTP GET requests to the Recoup API to check the sandboxStatus.

The sequence above shows the transition check for sandboxes, ensuring the system does not proceed until the ephemeral environment has successfully shut down.
Sources: src/pulse/pollSandboxUntilStopped.ts, src/pulse/tests/pollSandboxUntilStopped.test.ts:40-55

Social Media Scrape Batches

To manage external resource constraints, social media scraping is performed in chunks. The scrapeAndPollSocials function implements a batch-and-wait strategy.

Key Polling Utilities and Tasks

Content Run Polling

pollContentRunTask is a Trigger.dev task specifically designed to wait for the create-content task to complete. It has a high maxDuration (35 minutes) to account for video rendering times.

• ID: poll-content-run
• Logic: Executes pollContentRuns(), then calls sendContentCallback() to notify the main API.
• Error Handling: Resolves an "overall status" (completed, failed, or timeout) based on the combined results of all sub-runs.

Sources: src/tasks/pollContentRunTask.ts:11-35, src/content/tests/resolveOverallStatus.test.ts:7-30

Scraper Result Polling

The utility pollScraperResults manages the lifecycle of social media data collection. It differentiates between SUCCEEDED (returning data) and FAILED (returning error context).

// src/polling/pollScraperResults.ts:21-25
export async function pollScraperResults(
  runs: ScrapeRun[]
): Promise<PollResult[]> {
  const results: PollResult[] = [];
  const pendingRuns = new Map<string, ScrapeRun>(
    runs.map((run) => [run.runId, run])
  );
  // ... loop logic
}

Sources: src/polling/pollScraperResults.ts:21-30

Summary

Polling and async state hooks provide the glue between the task-based architecture and the RESTful APIs that consume them. By abstracting the complexity of state transitions—whether in Trigger.dev runs, Vercel Sandboxes, or third-party scrapers—these utilities ensure that the Recoup system remains eventually consistent and resilient to long-running task durations.