Express API & Process Manager
Relevant source files
The following files were used as context for generating this wiki page:
The Express API & Process Manager serves as the central backend orchestrator for the bot-auto-ig system. It is responsible for handling incoming HTTP requests from the frontend, managing authentication and rate limiting, and controlling the lifecycle of Python-based automation subprocesses. The system integrates with a Convex backend for persistent data storage and utilizes WebSockets for real-time progress updates.
The core responsibility of this module is to translate high-level automation requests into managed system processes while ensuring operational stability through features like mutex-based concurrency control, orphaned process cleanup, and state recovery after crashes.
Sources: server/index.ts:1-15, server/automation/routes.ts:1-25
System Architecture
The backend is built on Express and utilizes a modular routing structure. It acts as a bridge between the React-based frontend and the Python automation layer.
Core Components
- Express Server: Handles HTTP routing, middleware (Cerk auth, CORS, logging), and global error handling.
- Process Manager: Manages the spawning, monitoring, and killing of Python subprocesses. It tracks PIDs to prevent orphaned processes.
- Convex Client: A specialized HTTP client with exponential backoff and retry logic used to communicate with the Convex database.
- WebSocket Hub: Broadcasts real-time logs and status changes from automation processes to connected clients.
Sources: server/index.ts:40-100, server/shared/convexClient.ts:150-170, server/automation/routes.ts:100-130
Communication Flow
The following diagram illustrates how a user request flows through the API to trigger a managed process.
Process Management & Lifecycle
Process management is handled via the automationRouter and ProcessService. The system ensures that only one automation session (or a controlled number of workflow concurrent runs) occurs at a time using a mutex.
Subprocess Execution
Automation is triggered by spawning Python scripts using spawnPython from the ProcessService. The server pipes settings into the process via stdin and monitors stdout for structured log data.
| Feature | Implementation Detail | Source |
|---|---|---|
| Runner Script | python/runners/run_multiple_accounts.py | server/automation/routes.ts:21 |
| Concurrency Control | automationMutex (async-mutex) | server/automation/routes.ts:48 |
| Process Tracking | PID saved to disk and automationState.process | server/automation/routes.ts:114 |
| Graceful Shutdown | killProcess and signal handlers (SIGTERM/SIGINT) | server/automation/routes.ts:70, server/index.ts:140 |
Workflow Concurrency
For the workflow system, the API enforces a concurrency limit defined by the WORKFLOW_MAX_CONCURRENCY environment variable.
Database Integration (Convex Client)
The server communicates with Convex via a custom HTTP client. This client is designed to be resilient against transient network issues.
Resilience Strategy
- Retryable Errors: The client identifies
ECONNRESET,ETIMEDOUT, and HTTP 429/5xx codes as retryable. - Exponential Backoff: Retries use the formula:
baseDelay * 2^attempt + jitter. - Default Config: 3 retries with a 1000ms base delay.
Sources: server/shared/convexClient.ts:90-135
Key Data Models
The convexClient.ts defines several TypeScript types that mirror the database schema:
- DbProfileRow: Represents a browser profile (proxy, cookies, scraping limits).
- DbWorkflowRow: Represents a workflow definition (nodes, edges, schedule).
- DbWorkflowArtifactRow: Represents data scraped during a workflow (followers, storage IDs).
Sources: server/shared/convexClient.ts:40-80, server/shared/convexClient.ts:460-480
API Endpoints Summary
Automation & Workflows
| Endpoint | Method | Description |
|---|---|---|
/api/automation/start | POST | Starts the main multi-account automation runner. |
/api/automation/stop | POST | Terminates the running automation process. |
/api/workflows/run | POST | Executes a specific workflow by ID. |
/api/workflows/status | GET | Returns the current status of a workflow run. |
/api/workflows/artifacts | GET | Lists artifacts (scraped data) for a workflow. |
Sources: server/automation/routes.ts:35-80, server/workflows/routes.ts:30-100
Artifact Management
The API provides specialized routes for downloading large JSON artifacts stored in Convex. It fetches the private storage URL and streams the data to the client with appropriate headers.
// server/workflows/routes.ts:70-90
router.get('/artifacts/download', asyncHandler(async (req, res) => {
const url = await workflowArtifactsGetStorageUrl(storageId);
const upstream = await fetch(url);
const arrayBuffer = await upstream.arrayBuffer();
res.setHeader('Content-Type', 'application/json');
res.send(Buffer.from(arrayBuffer));
}));
Sources: server/workflows/routes.ts:71-88
Startup & Recovery Logic
When the server starts, it performs a series of maintenance tasks to ensure the system is in a clean state:
- Shutdown Handlers: Registers listeners for
SIGTERMandSIGINTto close the HTTP server and WebSockets gracefully. - Interrupted Run Detection: Checks for a PID file from a previous session. If found, it clears the stale state and logs a warning.
- Orphan Cleanup: Invokes
cleanupOrphanedProcesses()to kill any lingering Python processes from a previous crash. - Profile Reconciliation: Resets "busy" or "running" flags in the Convex database for browser profiles that are no longer associated with active PIDs.
Sources: server/index.ts:135-180, server/automation/process-manager.ts
Conclusion
The Express API and Process Manager provide a robust execution environment for the project's automation features. By combining strict concurrency controls, a resilient database client, and automated recovery procedures, the system maintains high availability even when dealing with unpredictable browser-based automation tasks. The separation of the management layer (Express) from the execution layer (Python) allows for scalable process handling and real-time observability.