Introduction to dbt Core v2 (Fusion)
Relevant source files
The following files were used as context for generating this wiki page:
dbt Core v2.0, also referred to as the foundation of the Fusion engine, represents a ground-up rewrite of the dbt Core architecture in Rust. This version is engineered for extreme performance at scale, significantly reducing parse, compile, and run times compared to the Python-based v1 implementation. It introduces a stricter language specification to enforce correctness at parse time and shifts toward a more scalable artifact format using Parquet.
Sources: README.md:9-25, crates/dbt-docs-server/README.md:14-16
Core Architectural Shifts
The transition from v1 to v2 (Fusion) introduces several fundamental changes to the internal engine and distribution model:
- Rust Implementation: The core engine is rewritten in Rust, eliminating the need for a Python runtime or virtual environments for local execution.
- Self-Contained Binary: dbt v2 is distributed as a single executable, simplifying installation and dependency management.
- Parquet Artifacts: Unlike v1, which primarily relied on
manifest.json, v2 produces Parquet artifacts. These artifacts are queryable, joinable, and designed to handle metadata for massive projects that exceed the practical limits of large JSON files. - ADBC Protocol: The engine uses the Arrow Database Connectivity (ADBC) protocol to submit queries to supported warehouses like Snowflake, BigQuery, and Postgres.
Sources: README.md:18-28, CONTRIBUTING.md:10-15, crates/dbt-docs-server/README.md:14-16
Tier Comparison: Core vs. Fusion
The richness of features depends on whether the user is running the standard dbt Core distribution or is signed into the Fusion engine.
| Feature | dbt Core v2 | Fusion Engine |
|---|---|---|
| License | Apache 2.0 | Extended (Closed source components) |
| Parsing | Fast Rust Parser | Fast Rust Parser |
| Metadata | Node-to-node lineage | Column-level lineage |
| Data Insights | Catalog and status | Sample data & inferred types |
Sources: README.md:55-61, crates/dbt-docs-server/README.md:51-56
Execution Lifecycle and Data Flow
The Fusion engine follows a structured resolution and execution path, moving from raw source files to queryable Parquet metadata and warehouse execution.
The diagram above illustrates how project files are processed by the Rust-based resolver to generate both warehouse commands and Parquet metadata indices.
Sources: crates/dbt-docs-server/README.md:25-36, crates/dbt-parser/src/resolver.rs:114-125
The Resolve Phase
The resolve function in the dbt-parser crate serves as the entry point for turning project source files (models, seeds, tests, macros) into a finalized ResolverState. Key responsibilities include:
- Macro Resolution: Resolving macros from all packages, including a default
__overview__for documentation. - Environment Initialization: Creating a Jinja environment (via
minijinja) configured with adapter-specific logic. - Topological Waves: Processing packages in waves based on dependency levels to ensure configuration properties propagate correctly.
- Validation: Checking for duplicate relations and validating function configurations against adapter capabilities (e.g., JS UDF support in BigQuery/Snowflake).
Sources: crates/dbt-parser/src/resolver.rs:107-160, crates/dbt-parser/src/resolver.rs:240-250
Scalable Documentation (dbt-docs-server)
v2 replaces the static dbt docs generate approach with a dynamic, high-performance server. The dbt-docs-server is a read-only, stateless HTTP server built with the Axum framework.
Document Server Architecture
The server loads Parquet files from <target>/index/ into an in-memory DuckDB instance using ADBC. It serves a React Single Page Application (SPA) that is baked directly into the Rust binary.
Sources: crates/dbt-docs-server/README.md:23-41
Development and Contributions
Contributing to dbt Core v2 requires the Rust toolchain. The project uses a workspace-based architecture with various crates handling specific responsibilities (e.g., dbt-parser, dbt-docs-server, dbt-schemas).
Environment Requirements
- Rust Toolchain: Required for building and testing.
- Nextest: The preferred testing utility for running the project's test suite.
- Database Drivers: Drivers for Snowflake, BigQuery, and Postgres are submitted via the ADBC protocol.
Sources: CONTRIBUTING.md:25-35, CONTRIBUTING.md:10-15
Build Commands
To build the project locally, developers use standard cargo commands:
# Debug build
cargo build
# Release build
cargo build --release
Sources: CONTRIBUTING.md:38-41
Conclusion
dbt Core v2 (Fusion) marks a significant evolution from its Python predecessor, prioritizing performance through Rust and scalability through Parquet-based metadata. By shifting to a self-contained binary and implementing a dynamic documentation server, it reduces the complexity of managing dbt environments while providing a more robust foundation for large-scale data engineering workflows.
Sources: README.md:18-35, crates/dbt-docs-server/README.md:14-18