Background Jobs & Retries
Relevant source files
The following files were used as context for generating this wiki page:
- backend/src/workflows/application/retry_service.py
- backend/src/workflows/application/action_executor.py
- backend/src/workflows/application/execution_service.py
- backend/src/workflows/domain/execution_entities.py
- backend/src/workflows/infrastructure/execution_models.py
- DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md
Background Jobs and Retries form the core execution engine of the workflow automation system. This system is responsible for processing sequential actions, managing long-running "wait" states, and ensuring system resilience through sophisticated error handling and exponential backoff strategies. The architecture is designed to handle asynchronous tasks such as sending emails, processing webhooks, and updating CRM records while maintaining a consistent state machine for every execution.
The system ensures that transient failures—such as network timeouts or external API rate limits—do not result in permanent workflow failure. By utilizing a specialized RetryService, the platform can automatically re-enqueue failed actions with increasing delays, maximizing the probability of successful completion without manual intervention.
Sources: DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:10-25, backend/src/workflows/application/execution_service.py
Workflow Execution Architecture
The execution engine follows a sequential processing model where each step of a workflow is treated as a discrete job. The lifecycle of these jobs is managed by the WorkflowExecutionService, which orchestrates the transition between different execution states.
Execution Lifecycle States
The system implements a 7-state lifecycle machine to track the progress of background tasks:
| State | Description |
|---|---|
| QUEUED | Initial state when a trigger creates an execution. |
| ACTIVE | The job is currently being processed by the engine. |
| PAUSED | Execution is manually suspended. |
| WAITING | Suspended state awaiting a time delay or external event. |
| COMPLETED | All actions in the workflow finished successfully. |
| FAILED | The job stopped due to an unrecoverable error or exhausted retries. |
| CANCELLED | Manually terminated by a user. |
Sources: backend/src/workflows/domain/execution_entities.py:20-40, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:140-160
Execution State Machine Flow
The following diagram illustrates the valid transitions between job states within the background engine.
This diagram shows the movement of a workflow execution through its lifecycle, specifically highlighting how failed jobs can return to the QUEUED state for retry.
Sources: DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:520-550
Action Execution & Background Jobs
Every step in a workflow is handled by an ActionExecutor. These executors are factory-generated components that translate domain configurations into side effects (e.g., sending an SMS or calling a Webhook).
Supported Background Task Types
The engine currently supports several specialized executors:
- Communication Executors:
SendEmailExecutor,SendSMSExecutor, andSendVoiceExecutor. - Logic & Timing:
WaitTimeExecutor, which calculatesresume_attimestamps to pause execution. - Integrations:
WebhookExecutor, which performs HTTP requests and parses rate-limit headers. - CRM Operations:
UpdateContactExecutor,AddTagExecutor, andRemoveTagExecutor.
Sources: backend/src/workflows/application/action_executor.py:50-150, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:200-240
Job Processing Logic
The WorkflowExecutionService handles the sequential advancement of steps. When an action finishes, the engine automatically enqueues the next step or marks the entire execution as COMPLETED.
Sources: backend/src/workflows/application/execution_service.py:100-120, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:380-395
Retry Mechanism & Resilience
The RetryService is the primary component for ensuring resilience against transient errors. It categorizes errors and applies a RetryStrategy to determine if and when a job should be attempted again.
Error Categorization
The system distinguishes between different failure types to decide on the appropriate recovery path:
| Error Category | Retry | Strategy |
|---|---|---|
| Timeout | Yes | Exponential Backoff |
| Rate Limit | Yes | Delay based on headers or backoff |
| Server Error (5xx) | Yes | Exponential Backoff |
| Network Error | Yes | Exponential Backoff |
| Validation (4xx) | No | Permanent Failure |
| Auth/Authz | No | Permanent Failure |
Sources: backend/src/workflows/application/retry_service.py:50-80, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:650-670
Exponential Backoff Strategy
The default RetryStrategy uses exponential backoff with jitter to prevent "thundering herd" problems during service outages.
- Max Attempts: 3 (Default)
- Base Delay: 60 seconds
- Exponential Base: 2
- Jitter: 10% (randomized delay variance)
Calculation Pattern: base_delay * (exponential_base ^ attempt_number)
Sources: backend/src/workflows/application/retry_service.py:10-30, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:680-700
Retry Execution Sequence
This sequence diagram shows how the WorkflowExecutionService interacts with the RetryService and ActionExecutor when a transient error occurs.
This diagram demonstrates the error handling flow where a timeout triggers a retry check and subsequent re-scheduling of the job.
Sources: backend/src/workflows/application/execution_service.py:150-180, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:580-620
Persistence & Audit Logs
Background jobs are persisted in the infrastructure layer to ensure state is maintained across system restarts.
Data Models
- WorkflowExecutionModel: Tracks the high-level state of the background job, including
current_step_index,retry_count, andstatus. - ExecutionLogModel: A detailed audit trail of every attempted action, capturing
duration_ms,error_details, andresponse_data.
Sources: backend/src/workflows/infrastructure/execution_models.py:10-50, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:250-280
Concurrency & Rate Limiting
The engine enforces a Concurrent Execution Limit (default: 100 per account) to prevent background jobs from overwhelming system resources. If this limit is exceeded, new jobs are rejected with a ConcurrentExecutionLimitError.
Sources: backend/src/workflows/application/execution_service.py:80-95, DDD_IMPLEMENTATION_REPORT_SPEC_WFL_005.md:180-195
Conclusion
The Background Jobs & Retries system provides the robust infrastructure necessary for reliable marketing automation. By combining a strict state machine with a flexible retry service and detailed execution logging, the system ensures that complex multi-step workflows are executed predictably, even in the face of external service instability. This architecture guarantees high availability and data integrity for all automated customer interactions.