Project & Macro Loader
Relevant source files
The following files were used as context for generating this wiki page:
The Project & Macro Loader is a critical system responsible for initializing the dbt environment by discovering, parsing, and validating project configuration files (dbt_project.yml), connection profiles, and Jinja macros. It acts as the bridge between the physical files on disk (or embedded assets) and the logical DbtState used by the compiler and execution engine.
This system handles the discovery of models, seeds, snapshots, and tests across the root project and its dependencies, while also managing the specialized loading of internal adapter macros required for database-specific operations. It supports both cold starts (full disk scans) and warm starts using incremental state caches to optimize performance.
Core Loading Architecture
The loading process follows a hierarchical sequence: resolving basic project identities, initializing the Jinja environment, installing or locating package dependencies, and finally performing a deep scan of all resource paths.
The entry point for this process is the load function, which orchestrates the transition from raw YAML to a structured DbtState.
Sources: crates/dbt-loader/src/loader.rs:188-210
Key Data Structures
| Structure | Description |
|---|---|
DbtState | The primary container for the loaded project, containing all packages, variables, and connection profiles. |
DbtPackage | Represents a single project unit (root or dependency), tracking its specific resource paths and assets. |
DbtAsset | A metadata pointer to a specific file (e.g., .sql or .yml) within a package. |
MacroAssets | A RustEmbed structure used to bundle internal adapter macros directly into the binary. |
Sources: crates/dbt-loader/src/loader.rs:271-282, crates/dbt-loader/src/load_packages.rs:31-38
Project Configuration & Profile Loading
The loader first processes the dbt_project.yml file to determine the project's identity, variable definitions, and resource paths. It utilizes a "simplified" project loader to extract mandatory flags and configurations before the full Jinja environment is ready.
Variable Precedence
Variables are merged with the following precedence (highest to lowest):
- CLI Arguments (
--vars) vars.yml(at the project root)dbt_project.ymldeclarations
Sources: crates/dbt-loader/src/loader.rs:596-613
Sequence of Configuration Loading
Macro Discovery and Resolution
Macros are loaded from two primary sources: internal adapter macros and user-defined macros in the project's macros/ directories.
Internal Package Loading
dbt requires specific internal macros (e.g., dbt-adapters, dbt-postgres) to function. These can be loaded in three modes:
- Embedded: Macros are read directly from the binary's
RustEmbedassets (no disk I/O). - ForceWrite: Embedded assets are synchronized to a hidden directory on disk using SHA-256 hash comparison to ensure they are up to date.
- ReadFromDisk: Existing macros in the installation path are used.
Sources: crates/dbt-loader/src/load_packages.rs:104-142, crates/dbt-loader/src/loader.rs:434-455
Macro Parsing Logic
The system identifies macros by scanning .sql or .jinja files for {% macro ... %}, {% test ... %}, and {% materialization ... %} blocks.
Asset Collection and Filtering
The loader performs a deep scan of all directories defined in dbt_project.yml (e.g., model-paths, seed-paths). It uses a sophisticated filtering mechanism to categorize files and respect ignore rules.
Resource Path Categorization
Assets are grouped by ResourcePathKind:
ModelPaths:.sql,.pySeedPaths:.csv,.parquet,.jsonMacroPaths:.sqlDocsPaths:.md
Sources: crates/dbt-loader/src/loader.rs:807-850
Ignore Logic
The loader respects .dbtignore files found in the project root. It uses a GitignoreBuilder to filter out files that should not be processed by the compiler.
Sources: crates/dbt-loader/src/loader.rs:885-918
Macro Patching and Validation
Once macros are parsed from SQL files, they are "patched" using metadata found in schema.yml files. This process attaches descriptions, meta-tags, and documentation to the macro objects.
Argument Validation
If validate_macro_args is enabled in project flags, the loader performs the following checks:
- Name Matching: Ensures argument names in YAML match the actual Jinja parameters.
- Type Checking: Validates that type strings (e.g.,
list[str],dict[str, int]) follow the supported grammar. - Inference: Automatically infers undocumented arguments from the Jinja definition.
Sources: crates/dbt-parser/src/resolve/resolve_macros.rs:245-290
Conclusion
The Project & Macro Loader is a high-performance subsystem that handles the complexities of dbt's file-based configuration and Jinja-heavy environment. By supporting embedded assets and incremental warm starts, it ensures that the project state is initialized quickly and accurately, providing a robust foundation for the subsequent resolution and compilation phases.