Data Uploader & Ingestion Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• datauploader/uploader.py
• datauploader/api.py
• datauploader/scraping_tasks.py
• frontend/src/features/accounts/hooks/useDataUploader.ts
• convex/schema.ts
• server/shared/convexClient.ts
• python/runners/workflow/scrape_relationships.py

The Data Uploader & Ingestion Pipeline is a specialized service designed to handle the ingestion, processing, and storage of large-scale social media data within the bot-auto-ig ecosystem. Its primary purpose is to bridge the gap between raw scraped data—captured as workflow artifacts—and the structured database environment (Convex). It provides mechanisms for uploading CSV files and importing artifacts generated by automated scraping tasks.

The pipeline manages the lifecycle of data from initial extraction by Python-based executors to final persistence in the cloud backend. It supports deduplication, field filtering, and environment-specific deployments (Dev/Prod), ensuring that scraped user profiles and relationship data (followers/following) are correctly mapped to the system's internal schemas.

Pipeline Architecture

The ingestion pipeline consists of three main stages: extraction, staging (artifacts), and ingestion. Data is typically extracted via Python executors, stored as JSON artifacts in cloud storage, and then ingested into the database through the Data Uploader API.

High-Level Data Flow

The following diagram illustrates how data moves from a scraping task to the final database storage.

Core Components

1. Scraping Artifact Management

When a workflow node (such as scrape_relationships) completes its execution, it generates a "Workflow Artifact." This artifact contains the raw list of scraped users, metadata about the run (node ID, kind, targets), and statistical summaries.

Field	Description
storageId	Unique ID of the raw JSON file in cloud storage.
kind	Type of data: followers or following.
stats	Statistics including scraped, deduped, and targetsCompleted.
imported	Boolean flag indicating if the data has been ingested into the main DB.

Sources: convex/schema.ts:167-195, server/shared/convexClient.ts:386-407

2. Data Uploader API

The Data Uploader service provides REST endpoints for managing the ingestion process. It allows the frontend to list pending scraping tasks, retrieve field schemas from those tasks, and trigger the final processing.

• List Tasks: Retrieves available artifacts for a specific environment.
• Import Task: Transitions an artifact into a processable state.
• Process Task: Executes the deduplication and ingestion logic into Convex.

Sources: frontend/src/features/accounts/hooks/useDataUploader.ts:30-100

Artifact Generation & Persistence Logic

The Python runner uses a dedicated executor, ScrapeRelationshipsExecutor, to manage the state of ongoing scrapes. It ensures data integrity through "Resume Snapshots," which prevent data loss during long-running tasks.

Persistence Sequence

Data Models & Schema

The ingestion pipeline interacts heavily with the workflowArtifacts and instagramAccounts tables. The workflowArtifacts table serves as a staging area, while instagramAccounts acts as the primary destination for processed data.

Workflow Artifact Schema

workflowArtifacts: defineTable({
    name: v.string(),
    workflowId: v.id("workflows"),
    kind: v.union(v.literal("followers"), v.literal("following")),
    targets: v.array(v.string()),
    storageId: v.optional(v.id("_storage")),
    stats: v.optional(v.object({
        scraped: v.number(),
        deduped: v.number(),
        chunksCompleted: v.number(),
        targetsCompleted: v.number(),
    })),
    imported: v.optional(v.boolean()),
})

Sources: convex/schema.ts:167-195

Ingestion Workflow (Frontend Integration)

The frontend provides a user interface for managing these uploads via the useDataUploader hook. This hook encapsulates the multipart file upload logic and the subsequent processing steps.

1. Selection: User selects a file or an existing scraping artifact.
2. Field Mapping: The API returns available fields (e.g., userName, fullName).
3. Processing: The user selects "Keep Fields" and chooses the target environment (dev or prod).
4. Completion: The service returns stats on the number of new records vs. duplicates.

Sources: frontend/src/features/accounts/hooks/useDataUploader.ts:135-180

Conclusion

The Data Uploader & Ingestion Pipeline provides a robust, fault-tolerant system for moving social data from the browser-based scraping environment into a structured database. By utilizing intermediate storage artifacts and a dedicated processing API, it ensures that large datasets are handled efficiently without blocking the main application backend.