Parsing & Compilation Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Parsing & Compilation Pipeline is the core engine of dbt-core (Fusion), responsible for transforming raw project source files—SQL models, Python scripts, YAML configurations, and Jinja macros—into a structured, executable Directed Acyclic Graph (DAG) known as the Manifest. This pipeline operates in distinct phases, beginning with the loading of project files, followed by a resolution phase where macros are expanded and configurations are merged, and concluding with a compilation/execution phase where SQL is rendered and run against the target database.
The pipeline is designed for high performance, utilizing a "Partial Parse" mechanism that allows it to skip unchanged files by comparing file checksums against a previously saved state. This is critical for maintaining developer productivity in large-scale projects with thousands of nodes. The final output of this pipeline is a ResolverState which contains a complete set of Nodes, Macros, and Operations ready for task scheduling and execution.
Sources: crates/dbt-parser/src/resolver.rs:105-115, crates/dbt-main/src/compilation.rs:125-135
Pipeline Overview and Data Flow
The pipeline executes in three primary stages: Load, Resolve, and Schedule/Run. The data flow is strictly governed by the project's dependency structure, ensuring that global components like macros are resolved before the models that reference them.
This diagram illustrates the high-level progression from raw files to the final execution manifest.
Sources: crates/dbt-main/src/compilation.rs:560-650, crates/dbt-parser/src/resolver.rs:118-200
Phase 1: Loading and Partial Parsing
The initial phase identifies all relevant files within the project's configured paths (e.g., model-paths, analysis-paths). The system uses DbtLoadedProject to manage this state. If --partial-parse is enabled, the pipeline attempts to load the CacheState. It compares the mtime and checksum of files to determine which nodes must be re-parsed.
Key Logic
- Discovery: Locates
dbt_project.ymland identifies package dependencies. - Caching: If the file set hasn't changed and a valid cache exists, the system can perform a "fast-load," skipping the expensive filesystem scan.
- Checksumming: Uses hashes of the raw source code to detect content changes.
Sources: crates/dbt-main/src/compilation.rs:160-205, crates/dbt-main/src/compilation.rs:815-840
Phase 2: The Resolution Phase
The resolve function in resolver.rs acts as the entry point for this phase. It manages the complex task of propagating configurations and extracting dependencies between nodes.
1. Macro Resolution
Macros are resolved first because they are needed for rendering node SQL. The pipeline iterates through all packages, collecting macros from .sql files and documentation from .md files. A default __overview__ macro is injected if not provided by the user.
Sources: crates/dbt-parser/src/resolver.rs:136-160
2. Package-Wave Processing
Packages are resolved in "waves" based on their topological dependency order. Intra-wave parallelization allows multiple independent packages to be resolved simultaneously.
Sources: crates/dbt-parser/src/resolver.rs:800-850
3. Resource-Specific Resolution
Each resource type has a dedicated resolution logic:
| Resource | Description | Source Logic |
|---|---|---|
| Models | Resolves SQL and Python models, applying versioning and constraints. | resolve_models.rs |
| Sources | Maps YAML definitions to database relations and configures freshness rules. | resolve_sources.rs |
| Analyses | Resolves SQL files in analysis paths, extracting refs without creating relations. | resolve_analyses.rs |
| Seeds | Resolves CSV files as static data nodes. | resolver.rs:610 |
| Tests | Resolves both data tests (SQL) and unit tests. | resolver.rs:720 |
Sources: crates/dbt-parser/src/resolver.rs:550-750
Node Resolution Logic: Models and Sources
The resolution of a specific node involves merging multiple layers of configuration: project-level defaults, YAML property overrides, and inline config() calls.
SQL vs. Python Model Resolution
SQL models are rendered using Jinja, allowing the SqlFileInfo to extract ref() and source() calls during the render. Python models, however, are processed via AST analysis (Abstract Syntax Tree) to extract similar dependencies without a full Python runtime during the parse phase.
Source and Freshness Resolution
Sources are defined in YAML. The pipeline merges source-level and table-level configurations. A critical part of source resolution is the loaded_at_field and loaded_at_query conflict check: these two fields are mutually exclusive.
Validation and Dependency Extraction
After all nodes are initially resolved, the pipeline performs final validations and completes the DAG linkage:
- Dependency Resolution: Calls
resolve_dependenciesto maprefandsourcenames to unique node IDs across the entire project. - Access Control: Checks if private or protected models are being accessed from unauthorized groups or packages.
- Relation Uniqueness: Ensures no two models, seeds, or snapshots resolve to the same database relation (database.schema.identifier).
- Function Validation: Validates user-defined functions (UDFs) against adapter capabilities (e.g., JavaScript support on Snowflake vs. BigQuery).
Sources: crates/dbt-parser/src/resolver.rs:310-360, crates/dbt-parser/src/resolver.rs:410-450
Conclusion
The Parsing & Compilation Pipeline is a highly optimized, multi-phase system that ensures project integrity through rigorous validation and configuration merging. By separating global macro resolution from node-specific resolution and utilizing topological waves for package processing, it maintains a scalable and reliable environment for transforming dbt project code into executable data pipelines. The final ResolverState serves as the definitive truth for the project's structure, enabling efficient task scheduling and execution.
Sources: crates/dbt-parser/src/resolver.rs:365-400, crates/dbt-main/src/compilation.rs:560-580