Data Scraping & Extraction
Relevant source files
The following files were used as context for generating this wiki page:
- frontend/src/features/scraped-data/containers/ScrapedDataPageContainer.tsx
- python/runners/workflow/scrape_relationships.py
- python/runners/workflow/scrape_navigation.py
- convex/schema.ts
- frontend/src/features/scraped-data/components/ScrapedDataDetails.tsx
- frontend/src/features/workflows/components/WorkflowDetails.tsx
- server/shared/convexClient.ts
The Data Scraping & Extraction system is a core component of the project designed to automate the retrieval of relationship data (followers and following lists) from Instagram. It operates as a distributed process involving a Python-based automation runner for browser interaction, a Convex backend for state management and data persistence, and a React frontend for monitoring and downloading the resulting artifacts.
The system is primarily orchestrated through workflows, where specific "scrape_relationships" nodes define the parameters for data extraction. The architecture emphasizes resilience through features like exponential backoff retries, daily scraping limits to prevent account flagging, and resume snapshots to handle interruptions.
Sources: python/runners/workflow/scrape_relationships.py:40-100, convex/schema.ts:161-193
Extraction Architecture
The extraction process is handled by the ScrapeRelationshipsExecutor, which manages the lifecycle of a scraping task. It coordinates between browser navigation, data chunk fetching, and state synchronization with the backend.
Extraction Logic Flow
The following diagram illustrates the high-level logic used by the Python runner to execute a scraping task:
The executor utilizes a "chunking" strategy, where data is fetched in small batches (controlled by chunkLimit and maxPagesPerAttempt) to minimize the impact of failures and stay within rate limits.
Sources: python/runners/workflow/scrape_relationships.py:72-108, python/runners/workflow/scrape_relationships.py:430-480
Key Components & Logic
Relationship Navigation
The system navigates to specific relationship views (Followers or Following) by identifying and clicking the appropriate links on a profile page. It uses a multi-layered approach:
- Selector Matching: Attempts to find links based on standard Instagram URL patterns (e.g.,
a[href="/username/followers/"]). - Locator Fallback: If selectors fail, it uses text-based locators searching for "Followers" or "Following" labels.
- UI Verification: Waits for the relationship dialog or a URL change to confirm the view is open before starting extraction.
Sources: python/runners/workflow/scrape_navigation.py:10-50
State & Resilience Management
The system maintains highly granular state to ensure that interrupted scrapes can be resumed without losing data or wasting account capacity.
| Feature | Description | File Reference |
|---|---|---|
| Resume Snapshots | Temporary local files storing scraped users and cursors for in-progress tasks. | scrape_relationships.py |
| Deduplication | Scraped users are passed through a deduplication filter (_dedupe_scraped_users) before storage. | scrape_relationships.py |
| Exponential Backoff | Configurable retry delays (default: 1, 2, 4 seconds) for transient errors or rate limits. | scrape_relationships.py |
| Daily Limits | Profiles track dailyScrapingUsed vs dailyScrapingLimit to prevent over-use. | schema.ts, convexClient.ts |
Sources: python/runners/workflow/scrape_relationships.py:138-160, server/shared/convexClient.ts:250-270
Data Persistence & Artifacts
Once a scraping task or a "kind" (followers/following) is completed, the system generates a workflowArtifact. This record links the extracted data stored in Convex file storage with metadata about the run.
Workflow Artifact Schema
The workflowArtifacts table in Convex stores the following data:
| Field | Type | Description |
|---|---|---|
name | string | Display name of the artifact. |
kind | "followers" | "following" | The type of relationship data extracted. |
targets | array | The usernames targeted for scraping. |
stats | object | Includes scraped, deduped, chunksCompleted, and targetsCompleted. |
storageId | id("_storage") | Reference to the raw extracted JSON data. |
manifestStorageId | id("_storage") | Reference to a manifest file summarizing the extraction. |
Sources: convex/schema.ts:161-193
Artifact Lifecycle
The lifecycle of data after extraction follows a strict path from the automation runner to the frontend:
Frontend Monitoring & Interaction
The ScrapedDataPage provides a centralized interface for managing extracted data. It allows users to:
- Search and Filter: Filter artifacts by name, workflow, node label, or target username.
- View Details: Detailed stats including row counts, completion status, and source profiles.
- Download: Direct download of raw data JSON and manifest files.
- Workflow Integration: Jump directly to the workflow editor from which an artifact originated.
Sources: frontend/src/features/scraped-data/containers/ScrapedDataPageContainer.tsx:40-70, frontend/src/features/scraped-data/components/ScrapedDataDetails.tsx:20-50
Summary
The Data Scraping & Extraction system provides a robust framework for harvesting Instagram relationship data. By combining browser-level automation with persistent state management and a comprehensive artifact system, it ensures data integrity while adhering to platform constraints through sophisticated rate limiting and retry logic.