Dependency Management & Packages
Relevant source files
The following files were used as context for generating this wiki page:
Dependency Management and Packages in dbt-core (v2/Fusion) facilitate the modularity and reusability of dbt code. The system allows projects to include external code—such as models, macros, and tests—from varied sources including the dbt Hub, Git repositories, local directories, and private registries. These dependencies are resolved and loaded into the project state during the parsing and resolution phases.
The system ensures that each dependency is correctly identified, and its configurations are merged with the root project's settings. By leveraging a topological wave-based resolution strategy, dbt-core manages complex inter-package dependencies while maintaining performance through parallel execution where safe.
Package Identification and Loading
The system identifies dependencies by scanning specific configuration files within the project root. The two primary files used for this purpose are packages.yml and dependencies.yml.
Dependency Source Types
Packages can be defined in several formats, each requiring a unique resolution logic:
- Hub: Packages hosted on the official dbt Hub.
- Git: Packages sourced directly from a Git repository, optionally specifying a subdirectory.
- Local: Packages located on the local filesystem relative to the root project.
- Private: Packages from private registries.
- Tarball: Packages distributed as compressed archives.
Sources: crates/dbt-loader/src/utils.rs:6-10, crates/dbt-loader/src/utils.rs:136-173
The Loading Flow
The identify_package_dependencies function coordinates the discovery of these files and utilizes a package_lookup_map (typically derived from package-lock.yml) to ensure consistency between defined and installed versions.
Topological Resolution Strategy
During the resolve phase, dbt-core processes packages in "waves." These waves are determined by the dependency graph, ensuring that a package's dependencies are always resolved before the package itself.
Wave-based Execution
- Topological Preparation: Packages are grouped into levels (waves) where level 0 contains packages with no dependencies, and subsequent levels contain packages depending only on previous levels.
- Intra-wave Parallelism: Within a single wave, packages are resolved in parallel using
dispatch_maybe_parallelto maximize CPU utilization. - Inter-wave Sequentiality: The system waits for all packages in a current wave to finish before starting the next wave to maintain data integrity.
Sources: crates/dbt-parser/src/resolver.rs:163-166, crates/dbt-parser/src/resolver.rs:595-635
Data Flow during Resolution
Each package resolution produces a set of nodes, macros, and runtime configurations which are then merged into the global ResolverState.
| Component | Responsibility |
|---|---|
resolve_package | Resolves a single package, creating its DbtRuntimeConfig. |
resolve_inner | Handles the internal logic of parsing models, seeds, and sources within a package. |
all_runtime_configs | A BTreeMap storing full runtime configurations for every package in the project. |
Sources: crates/dbt-parser/src/resolver.rs:530-580, crates/dbt-parser/src/resolver.rs:160-162
Configuration Merging and Overrides
A critical aspect of dependency management is the hierarchical application of configurations. dbt-core ensures that the root project can override configurations defined within dependency packages.
Merging Logic
Configurations are merged in a specific order of precedence:
- Local Project Config: Settings within the package's own
dbt_project.yml. - Root Project Config: Settings from the main project's
dbt_project.yml(overrides local). - Schema YAML Config: Settings defined in
.ymlfiles within the package. - Table/Node Level Config: Specific settings on individual resources.
Sources: crates/dbt-parser/src/resolve/resolve_sources.rs:88-100, crates/dbt-parser/src/resolve/resolve_sources.rs:245-257
Implementation Example: Sources
When resolving sources, the system specifically merges source-level and table-level configs before applying root overrides.
Package State and Metadata Persistence
To improve performance, dbt-core v2 utilizes a Parquet-backed incremental parse cache. This system stores package-level metadata and file timestamps to detect changes without full project re-parsing.
Metadata Layout
The metadata is stored in the target/metadata/parse/ directory.
| File | Description |
|---|---|
filestamps.parquet | Records per-package file paths and mtime_ns timestamps. |
project.parquet | Project-level key/value pairs including dependencies and profile hashes. |
resolver_state.parquet | Stores JSON-encoded package dependencies (pkg_deps_json) and kinds (pkg_kinds_json). |
Sources: crates/dbt-metadata/src/parse_state.rs:13-22, crates/dbt-metadata/src/parse_state.rs:114-135
Internal vs. External Packages
The metadata system distinguishes between different package "lifetimes":
- User Packages: Standard dependencies that are snapshotted and persisted.
- Internal Packages: Embedded in the binary; excluded from the Parquet cache to prevent redundant loading and double-add bugs.
- Inline Packages: Transient packages (e.g., for
--inlineSQL) that are never persisted.
Sources: crates/dbt-metadata/src/parse_state.rs:777-798
Conclusion
Dependency Management in dbt-core provides a robust framework for project composition. By combining structured package identification, topological resolution waves, and a sophisticated configuration merging hierarchy, the system allows large-scale data projects to remain modular and maintainable while ensuring root-level control over all sub-components.