Workflow Execution Flow

Relevant source files

The following files were used as context for generating this wiki page:

• packages/core/src/workflow/Workflow.ts
• packages/core/src/node/WorkFlowNode.ts
• packages/core/src/messages/pipeline/InvocationPipeline.ts
• apps/web/src/app/(protected)/workflows/page.tsx
• apps/web/src/lib/workflows.ts
• packages/core/src/utils/persistence/README.md
• packages/mcp-server/src/index.ts
• README.md

The Workflow Execution Flow is the core process that orchestrates the execution of a Directed Acyclic Graph (DAG) of AI agents, known as nodes. This system manages the entire lifecycle from workflow instantiation and preparation to the invocation of individual nodes, message passing, and persistence of results. The primary goal is to execute a series of tasks defined in a WorkflowConfig to achieve a high-level objective.

Execution can be triggered programmatically via the SDK, through the web interface, or via the MCP (Model Context Protocol) server. Each execution is tracked as a WorkflowInvocation in the database, with each step performed by a node recorded as a NodeInvocation. This allows for detailed observability, debugging, and performance analysis.

High-Level Execution Architecture

The execution begins with a Workflow instance, which is prepared with specific inputs (WorkflowIO). The run method orchestrates the execution, which involves invoking a sequence of WorkFlowNode instances starting from the entryNodeId. Each node invocation is managed by a three-phase InvocationPipeline that handles context preparation, AI model execution, and result processing.

The following diagram illustrates the high-level flow from triggering a workflow to its completion.

Workflow Instantiation and Preparation

Before a workflow can be run, it must be instantiated and prepared. This setup process involves verifying the configuration, registering the workflow version in the database, and preparing the inputs.

1. Workflow Creation

A Workflow object is created using the static Workflow.create() method. This method takes a WorkflowConfig, evaluation inputs, and optional context for evolutionary processes.

Sources: packages/core/src/workflow/Workflow.ts:121-140

2. Workflow Preparation

The workflow.prepareWorkflow() method is called to process the initial EvaluationInput and generate the WorkflowIO[] array, which contains the specific inputs for each run. This step also sets the final goal for the workflow. If persistence is enabled, this I/O data is updated in the WorkflowVersion table.

Sources: packages/core/src/workflow/Workflow.ts:153-169

3. Invocation Record Creation

For each WorkflowIO, a unique WorkflowInvocation is created in the database via workflow.createInvocationForIO(). This generates a workflow_invocation_id and records the initial state, linking it to a specific workflow_version_id.

This diagram shows the sequence for preparing a workflow for execution.

Execution Trigger

A workflow run can be initiated from multiple sources.

UI-Driven Execution

From the web interface, a user can click "Run" on a workflow. This triggers the handleRun function, which prompts for user input and calls the useInvokeWorkflow hook. This hook sends a POST request to the /api/workflow/run or a similar endpoint.

// Source: apps/web/src/app/(protected)/workflows/page.tsx:180-218
const handleRun = async (workflow: WorkflowWithVersions) => {
    if (!workflow.activeVersion) return;

    const userInput = prompt("Enter input for the workflow:");
    if (!userInput) return;

    setRunningWorkflows(prev => new Set(prev).add(workflow.wf_id));

    try {
      const result = await invokeWorkflow.mutateAsync({
        workflowVersionId: workflow.activeVersion.wf_version_id,
        evalInput: {
          type: "prompt-only",
          workflowId: workflow.wf_id,
          goal: userInput,
        },
      });
      // ... handle result
    } catch (error) {
      // ... handle error
    } finally {
      setRunningWorkflows(prev => {
        const next = new Set(prev);
        next.delete(workflow.wf_id);
        return next;
      });
    }
};

Sources: apps/web/src/app/(protected)/workflows/page.tsx:180-218

Programmatic Execution (SDK)

The Workflow.run() method is the primary entry point for programmatic execution. It orchestrates the entire run by calling runAllIO, which iterates through the prepared WorkflowIO and executes the workflow for each one.

// Source: packages/core/src/workflow/Workflow.ts:285-307
async run(options?: { onProgress?: WorkflowEventHandler; abortSignal?: AbortSignal }): Promise<RS<RunResult[]>> {
    throwIf(this.evaluated, "Workflow has already been evaluated")
    throwIf(this.hasRun, "Workflow has already been run")

    lgg.log(`[Workflow.run] Starting setup for ${this.getWorkflowVersionId()}`)
    await this.setup()

    lgg.log(`[Workflow.run] Setup complete, starting runAllIO for ${this.getWorkflowVersionId()}`)
    const { data: runResults, error } = await runAllIO(this, options)
    this.hasRun = true
    if (error) {
      lgg.error(`[Workflow.run] runAllIO failed for ${this.getWorkflowVersionId()}: ${error}`)
      return R.error(error, 0)
    }
    lgg.log(
      `[Workflow.run] runAllIO succeeded for ${this.getWorkflowVersionId()}, got ${runResults?.length || 0} results`,
    )
    this.runResults = runResults
    return R.success(runResults ?? [], 0)
}

Sources: packages/core/src/workflow/Workflow.ts:285-307

MCP Server Execution

Workflows can also be invoked via the MCP server, which exposes a lucky_run_workflow tool. This tool makes a POST request to the /api/v1/invoke endpoint.

Sources: packages/mcp-server/src/index.ts:168-245

Node Invocation

The execution of each node is managed by the WorkFlowNode class. The invoke method is the entry point for a node's execution.

The WorkFlowNode Class

The WorkFlowNode class is the concrete implementation of an agent in the workflow. It manages its own configuration, tools, and persistence.

This diagram shows the main components of the WorkFlowNode class.
Sources: packages/core/src/node/WorkFlowNode.ts:31-70

The InvocationPipeline

The WorkFlowNode.invoke method uses an InvocationPipeline to structure the execution into three distinct phases. This ensures a clean separation of concerns for context setup, AI interaction, and result processing.

// Source: packages/core/src/node/WorkFlowNode.ts:184-215
public async invoke(context: NodeInvocationCallContext): Promise<NodeInvocationResult> {
    const startTime = Date.now();
    emitAgentStart(this.nodeId, this.config.description);

    try {
      // ...
      const pipeline = new InvocationPipeline(enhancedContext, this.toolManager, true);

      // Execute pipeline steps sequentially
      await pipeline.prepare();
      await pipeline.execute();
      const result = await pipeline.process();
      // ...
      return result;
    } catch (error) {
      // ...
      throw error;
    }
}

The pipeline's phases are:

Phase	Class Method	Description
1. Prepare	pipeline.prepare()	Sets up the execution context, selects the tool strategy, and builds the initial messages to be sent to the AI model.
2. Execute	pipeline.execute()	Enters a loop to interact with the AI model. It sends messages, handles tool calls, and manages retries until a final response is generated.
3. Process	pipeline.process()	Takes the final AI response, validates it, generates a summary, determines the next node (handoff), and persists the results.

Sources: packages/core/src/node/WorkFlowNode.ts:184-215, packages/core/src/messages/pipeline/InvocationPipeline.ts, packages/core/README.md

This diagram shows the internal flow of the InvocationPipeline.

Persistence and State Management

The state of each workflow and node invocation is meticulously tracked in a database. This is crucial for observability, debugging, and the evolutionary capabilities of the system. The persistence layer abstracts the database interactions.

Database Schema

The following tables are central to tracking workflow execution:

Table	Description
Workflow	Stores the base definition of a workflow, including its ID and description.
WorkflowVersion	Contains immutable, versioned configurations of a workflow, including the JSON DSL and genealogy data (parent versions) for evolution.
WorkflowInvocation	Records an individual execution of a specific WorkflowVersion. It tracks the status (e.g., running, completed), input/output, and metadata.
NodeVersion	Stores the configuration of a node for a specific WorkflowVersion.
NodeInvocation	Records an individual execution of a NodeVersion within a WorkflowInvocation. It logs the summary, cost, status, and files used.

Sources: packages/core/src/utils/persistence/README.md

This ER diagram shows the relationships between the core execution tables.

Conclusion

The workflow execution flow in lucky is a robust and observable system for running complex, multi-agent tasks. It is designed for reliability and evolution, with a clear separation of concerns between workflow orchestration (Workflow), node-level execution (WorkFlowNode), and the three-phase InvocationPipeline. Every step is persisted to a database, providing a complete audit trail that is essential for debugging, analysis, and the platform's self-improving capabilities. The architecture supports various triggers, from a rich web UI to programmatic SDK and MCP server calls, making it a flexible and powerful execution engine.