Error Handling, Retries & Recovery

Relevant source files

The following files were used as context for generating this wiki page:

• server/shared/errors.ts
• server/shared/convexClient.ts
• frontend/src/lib/api.ts
• server/index.ts
• server/automation/routes.ts
• frontend/src/features/workflows/utils/workflowImportExport.ts

The bot-auto-ig project implements a multi-layered error handling and recovery strategy spanning the frontend, backend server, and database communication layers. This system ensures that transient network failures are mitigated through automated retries with exponential backoff, while persistent application-level errors are captured, categorized, and reported through structured JSON responses.

A critical component of this architecture is the recovery of automation states following server restarts or process interruptions. The system identifies orphaned processes and stale runtime flags to ensure the platform returns to a consistent state without manual intervention.

Structured Error Hierarchy

The server utilizes a centralized error hierarchy for Express route handlers. By throwing typed subclasses of the base AppError, the system ensures that the global error middleware translates exceptions into consistent API responses.

Error Classes and Categorization

Class	Status Code	Error Code	Description
NotFoundError	404	NOT_FOUND	Requested resource does not exist.
ValidationError	400	VALIDATION_ERROR	Input data fails schema or logic validation.
ConflictError	409	CONFLICT	State conflict (e.g., resource already exists).
ExternalServiceError	503	EXTERNAL_SERVICE_ERROR	Dependency service (like Convex) is unavailable.

Sources: server/shared/errors.ts:12-51

The global error-handling middleware in server/index.ts captures these errors, logs them using a structured logger, and returns a standardized payload:

{
  "success": false,
  "error": {
    "code": "ERROR_CODE",
    "message": "Descriptive message"
  }
}

Sources: server/index.ts:114-131, server/shared/errors.ts:60-72

Network Retry Mechanisms

Retries are implemented at two main boundaries: the Frontend-to-Server API calls and the Server-to-Convex Database calls. Both use exponential backoff with jitter to prevent thundering herd problems.

Database (Convex) Retry Logic

The convexFetch utility implements a retry loop specifically for transient network issues or rate-limiting from the Convex cloud.

Retryable conditions include HTTP 429 (Too Many Requests), HTTP 5xx (Server Errors), and specific Node.js network codes such as ECONNRESET, ETIMEDOUT, and ENOTFOUND.
Sources: server/shared/convexClient.ts:79-130, server/shared/convexClient.ts:161-193

Frontend API Retry Logic

The frontend apiFetch utility defaults to 3 retries for idempotent methods (GET, HEAD, OPTIONS). Mutating requests (POST, PUT, DELETE) do not retry by default to prevent duplicate operations unless explicitly opted-in.
Sources: frontend/src/lib/api.ts:25-50

Automation Recovery & State Reconciliation

To maintain system integrity, the server performs a reconciliation routine upon startup. This process handles scenarios where the server may have crashed while automation was active.

1. Interrupted Run Detection: The system checks for persistent state markers from the previous session. If an interrupted PID is found, it verifies if the process is still alive and clears the stale state.
2. Orphan Cleanup: Any Python processes remaining from previous runs that are no longer tracked are terminated.
3. Profile Status Reconciliation: The profileManager resets "running" or "busy" flags for profiles that were left in an inconsistent state due to an unexpected restart.

Sources: server/index.ts:140-184, server/automation/routes.ts:141-155

Validation & Edge Case Handling

Workflow Import Validation

The system performs rigorous validation on imported workflow JSON files to prevent corrupted states from entering the database.

The validation differentiates between "hard failures" (e.g., duplicate node IDs, missing start nodes, unknown activity IDs) and "warnings" (e.g., references to list IDs that do not exist in the current environment).
Sources: frontend/src/features/workflows/utils/workflowImportExport.ts:114-162, frontend/src/features/workflows/utils/workflowImportExport.ts:245-300

Automation Process Monitoring

When an automation process is spawned, the server wires lifecycle events to handle unexpected crashes. If the Python process exits with a non-zero code, the server broadcasts an error log via WebSockets and marks the automation status as idle.
Sources: server/automation/routes.ts:158-185

Summary

The error handling and recovery system is designed for resilience. By combining structured API errors, aggressive retry logic for transient network failures, and proactive state reconciliation during server startup, the project ensures that automation workflows are reliable and that the system can self-heal from common failure modes.