Automation Modules: Engagement & Following
Relevant source files
The following files were used as context for generating this wiki page:
The Automation Modules for Engagement and Following provide the core logic for interacting with social relationships on the platform. These modules handle complex tasks such as scraping followers/following lists, managing follow/unfollow actions, and maintaining the state of these operations across multiple accounts. The system is designed to be highly resilient, incorporating retry logic, daily limits, and state persistence to recover from interruptions.
The engagement system is integrated into a larger workflow engine that allows users to schedule and monitor automation tasks. It interfaces with a Convex-based backend for data storage and a React-based frontend for real-time monitoring and configuration. Key functionalities include relationship scraping with cursor-based pagination and automated login session management.
Relationship Scraping System
The relationship scraping system is responsible for retrieving lists of followers or following for targeted accounts. It utilizes a dedicated executor that manages the lifecycle of a scraping task, including target preparation, state hydration, and artifact generation.
Scrape Relationships Executor
The ScrapeRelationshipsExecutor class encapsulates the logic for scraping. It supports different "kinds" of scrapes (followers, following, or both) and can resolve targets either from explicit configuration or by identifying accounts currently marked for scraping in the database.
The scraping process is iterative and saves "resume snapshots" to allow recovery if the process is interrupted. Once all targets are exhausted, it generates a WorkflowArtifact which is stored in the backend for user download or further automation steps.
Sources: python/runners/workflow/scrape_relationships.py:34-118, python/runners/workflow/scrape_relationships.py:535-590
Scraping Configuration Options
| Parameter | Type | Default | Description |
|---|---|---|---|
kind | string | 'followers' | Options: 'followers', 'following', 'both'. |
chunkLimit | number | 200 | Maximum users to fetch per API call (Cap: 5000). |
maxPagesPerAttempt | number | 3 | Maximum scrolls/pages to fetch in one run. |
maxAttempts | number | 4 | Max retries for a single target on failure. |
useAccountUsernames | boolean | false | If true, scrapes accounts in 'scraping' status. |
Sources: python/runners/workflow/scrape_relationships.py:53-65
Engagement Actions: Follow & Unfollow
Engagement actions are high-level tasks performed by the automation runner. These actions are typically part of a broader session where multiple profiles are processed sequentially.
Action Execution Flow
The following sequence diagram illustrates how the server triggers an automation run, which then executes engagement actions.
The server uses a mutex to ensure only one automation process runs at a time. It communicates with the Python runner via stdin (for settings) and stdout (for real-time events).
Sources: server/automation/routes.ts:40-66, server/shared/convexClient.ts:430-460
Daily Limits and Health
Engagement actions are strictly governed by daily scraping limits and profile health metrics. Before performing an action, the system checks the daily_scraping_used versus the daily_scraping_limit assigned to a profile.
- Lease Management: Profiles are "leased" to workers to prevent concurrent access by multiple runners.
- Health Tracking: Successes and failures are recorded to adjust the
scrape_healthscore of a profile.
Sources: server/shared/convexClient.ts:40-60, server/shared/convexClient.ts:462-500
State and Persistence
To ensure reliability, the system maintains state at multiple levels: transient memory, local files, and the remote database.
Local State Recovery
The server persists the current automation status (PID, start time, settings) to automation_state.json. Upon restart, the handleInterruptedRun function detects if a process was previously running and performs cleanup if the process is no longer active.
Artifact Generation
When a scraping module completes its task, it doesn't just return data; it creates a WorkflowArtifact. These artifacts include:
- Storage IDs: References to files in Convex storage.
- Stats: Number of users scraped, deduped, and total targets completed.
- Manifests: Metadata describing the scrape parameters.
Sources: python/runners/workflow/scrape_relationships.py:650-680, frontend/src/features/workflows/components/WorkflowDetails.tsx:290-320
Monitoring and Feedback
Real-time feedback is provided to the user through a WebSocket connection. The system emits structured events such as profile_started, task_progress, and profile_completed.
Event Structure
Events are emitted as JSON strings over stdout by the Python runner, parsed by the server, and broadcast to the frontend.
| Field | Description |
|---|---|
type | The event category (e.g., 'log', 'task_progress'). |
profileName | The name of the profile currently performing the action. |
scraped | Current count of items processed in the active task. |
deduped | Unique count of items after removing duplicates. |
status | Current state of the workflow (running, idle, stopping). |
Sources: server/automation/routes.ts:115-140, python/runners/workflow/scrape_relationships.py:590-610
The Automation Modules for Engagement and Following ensure that high-volume interactions are performed safely and predictably. By combining granular control over browser actions with robust state management and real-time monitoring, the system provides a professional-grade platform for relationship automation.