Background Jobs & Workers
Relevant source files
The following files were used as context for generating this wiki page:
The Background Jobs & Workers system in Flow is responsible for handling long-running, asynchronous tasks that should not block the main REST API or user interface. This system facilitates complex operations such as automated agent evaluation, scheduled task execution, and high-volume data ingestion. Built on a stack of Redis and ARQ, it ensures that resource-intensive processes like LLM-based evaluations and prompt rewriting are managed efficiently.
Sources: README.md:3-5, README.md:120-125
Infrastructure & Technology Stack
The background processing architecture relies on a distributed worker model. The system uses Redis as the message broker and state store for job queues, while the worker service executes the defined functions.
| Component | Technology | Role |
|---|---|---|
| Broker | Redis | Message queue and job state persistence |
| Worker | ARQ Worker | Asynchronous task execution environment |
| Persistence | PostgreSQL | Long-term storage for job results and audit logs |
Sources: README.md:4-6, README.md:48-55
The following diagram illustrates the relationship between the API, the Redis broker, and the background workers.
Key Asynchronous Processes
Automated Evaluation Loop
The most critical background process is the nightly auto_eval_tick cron job. This job automates the continuous improvement of agents by running evaluations against "Golden Sets" and potentially triggering prompt rewrites.
The logic follows a multi-stage pipeline:
- Golden Set Evaluation: Runs the current agent version against test cases using an LLM judge.
- Failure Analysis: If the pass rate falls below 0.7, failed items are sent to the Prompt Rewriter.
- Candidate Generation: The Rewriter (GPT-4o-mini) proposes edits; if confidence is high (≥ 0.3), a new
CANDIDATEgenome is created. - A/B Testing: The candidate is tested against the baseline.
- Human Approval: If successful, a proposal is created for human review in the UI.
Sources: README.md:120-142, services/api/flow/interfaces/http/routes/golden_sets.py:315-330
Sources: README.md:120-138
Scheduled Agent Executions
Agents can be configured with cron_expr strings (e.g., 0 8 * * *) to run tasks automatically. These schedules are stored in the agent_schedules table and executed by the worker.
Schedule Data Model Fields:
cron_expr: Standard cron timing instruction.prompt_template: The specific prompt to use for the run.delivery_type/delivery_target: Configuration for routing the agent's output.
Sources: README.md:155-163, apps/web/src/app/(app)/dashboard/page.tsx:96-100
Data Ingestion & Knowledge Syncing
Background workers handle the embedding and chunking of uploaded documents and crawled URLs. This process ensures that knowledge retrieval is available for agents without stalling the upload UI.
Sources: README.md:65-80, apps/web/src/app/(app)/knowledge/page.tsx:125-140
Data Persistence & Tracking
Execution results and background task statuses are tracked in PostgreSQL to provide observability in the System Dashboard.
| Table | Description |
|---|---|
executions | Tracks the status (completed, failed, running) and timing of jobs. |
execution_events | Stores granular payloads for events occurring during a background run. |
agent_schedules | Stores the definitions for recurring background agent tasks. |
golden_results | Stores the output and scores of background evaluation runs. |
Sources: apps/web/src/app/(app)/dashboard/page.tsx:80-105, services/api/flow/interfaces/http/routes/golden_sets.py:230-245
The executions table is essential for the "Recent Executions" view in the dashboard, capturing both status and error details for background tasks.
# From apps/web/src/app/(app)/dashboard/page.tsx:88-95
{
"table": "executions",
"columns": [
"id",
"agent_id",
"workspace_id",
"status",
"error",
"user_message",
"created_at",
"completed_at",
],
}
Sources: apps/web/src/app/(app)/dashboard/page.tsx:88-95
Conclusion
The Background Jobs & Workers system provides the foundational automation required for Flow's self-improvement cycle. By offloading heavy LLM processing, evaluation loops, and scheduled crons to a dedicated worker layer via Redis and ARQ, the system maintains high availability and responsiveness while continuously refining agent performance.