DAG Scheduling & Task Execution
Relevant source files
The following files were used as context for generating this wiki page:
DAG Scheduling and Task Execution represent the core engine logic responsible for transforming a project's resource definitions into an actionable sequence of work. This system manages the transition from a static dependency graph to a dynamic execution plan, handling node selection, dependency resolution, and multi-threaded task orchestration.
The process is divided into three primary phases: Scheduling, where nodes are filtered and sorted based on CLI arguments; Graph Construction, where nodes are mapped to specific tasks (Render, Analyze, Run); and Execution, where a visitor pattern coordinates the parallel or sequential processing of these tasks across worker threads.
Sources: crates/dbt-main/src/compilation.rs, crates/dbt-scheduler/src/schedule.rs
Architecture Overview
The scheduling and execution pipeline utilizes a phased approach to move from the initial ResolverState to a finalized set of RunTaskResults.
The architecture separates the logical selection of nodes from the physical orchestration of tasks. This allows the system to support complex behaviors like "fail-fast," incremental retries, and sidecar execution boundaries.
Sources: crates/dbt-tasks-sa/src/graph.rs:88-150, crates/dbt-main/src/compilation.rs:880-950
Node Scheduling and Selection
The build_schedule function in the scheduler module is the entry point for determining which nodes in the manifest should participate in the current invocation. It interprets CLI selectors (e.g., --select, --exclude) and handles indirect selection for tests.
Selection Logic
Scheduling involves several layers of filtering:
- Direct Selection: Expanding atoms from the
ResolvedSelectorinto a set ofunique_ids. - Indirect Selection: Determining which tests to include based on their parents (Eager, Cautious, or Buildable modes).
- Greedy Exclusion: If any parent of a test is explicitly excluded, the test itself is excluded, regardless of other selection criteria.
- Resource Type Filtering: Restricting the schedule to specific types like models, seeds, or snapshots.
Sources: crates/dbt-scheduler/src/schedule.rs:136-220, crates/dbt-scheduler/src/schedule.rs:250-280
Multi-source BFS
For graph operators like + (upstream) and + (downstream), the scheduler implements a multi-source Breadth-First Search (BFS). This allows it to follow edges for a specified max_depth from all initially selected nodes simultaneously, reducing complexity from $O(S \times (V+E))$ to $O(V+E)$.
Sources: crates/dbt-scheduler/src/schedule.rs:770-805
Task Graph Construction
Once the Schedule is finalized, the GraphBuilder transforms the selected nodes into a DiGraph of Task objects. Unlike the node DAG, the Task Graph is "phased," meaning a single node may be represented by multiple tasks (e.g., a Render task followed by an Analyze task followed by a Run task).
Execution Phases
Tasks are categorized into phases defined by the TP enum:
| Phase | Description |
|---|---|
| Render | Jinja rendering of SQL or configuration strings. |
| Analyze | Static analysis and type checking of the rendered SQL. |
| Run | Physical execution of the code against the target adapter. |
| Show | Previewing data for the dbt show command. |
Sources: crates/dbt-tasks-sa/src/graph.rs:38-55
Dependency Propagation
The Task Graph ensures strict ordering between these phases. For example, a model's Run task depends on its own Analyze task, which in turn depends on its Render task. Furthermore, dependencies between nodes are maintained; if Model B depends on Model A, then Model B (Render) will depend on Model A (Run) or Model A (Analyze) depending on the static analysis level.
Task Execution & The Visitor Pattern
The visitor module handles the physical execution of the Task Graph. It supports both sequential and parallel execution modes, utilizing a worker slot pool to manage concurrency.
Worker Slot Management
A WorkerSlotPool manages reusable logical worker IDs (Thread-1, Thread-2, etc.). Tasks acquire a slot before execution and release it upon completion. These IDs are used for telemetry and accessible within Jinja contexts via the thread_id variable.
Sources: crates/dbt-tasks-sa/src/visitor.rs:34-62
Parallel Execution Logic
The visit_parallel function implements an asynchronous loop that:
- Identifies "ready" nodes (those with an in-degree of 0).
- Spawns tasks using
tokio::spawnup to the concurrency limit. - Listens for completion events on a channel.
- Decrements the in-degree of dependent tasks once their parent succeeds.
Failure Propagation
When a task fails, the SkipSet determines which downstream tasks must be skipped. Failures are propagated DFS-style through the Task Graph. If fail-fast is enabled, the visitor immediately triggers a global shutdown. If not, it continues executing independent branches of the graph.
Sources: crates/dbt-tasks-sa/src/visitor.rs:88-147, crates/dbt-tasks-sa/src/visitor.rs:200-230
Retries and Incremental Execution
The system supports retrying failed invocations by reading previous run_results.json files. The prepare_for_potential_retry function modifies the EvalArgs and creates a DbtCustomScheduleDescription that specifically selects nodes that previously failed or were skipped.
During a retry, previous_batch_results are passed into the TaskRunner to avoid re-executing successfully completed overloads in microbatch models or multi-batch tests.
Sources: crates/dbt-main/src/dbt_lib.rs:410-475, crates/dbt-main/src/compilation.rs:880-900
Summary
The DAG Scheduling and Task Execution system in dbt-core provides a robust, multi-threaded environment for processing complex data transformation workflows. By decoupling node selection from phased task orchestration and using a visitor pattern for execution, the engine achieves high performance and reliability while maintaining strict dependency integrity. Significance lies in its ability to handle large-scale graphs with thousands of nodes, providing granular control over rendering, analysis, and run phases.