Adapter Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Adapter Architecture in dbt-core (Fusion engine) serves as the translation layer between dbt's generic transformation logic and the specific dialect and API requirements of various data warehouses. It provides a unified interface for executing SQL, managing database connections, and introspecting metadata across supported platforms such as Snowflake, BigQuery, Databricks, Redshift, and DuckDB.
The system is designed as a multi-layered facade. The outermost layer is the Adapter struct, which integrates with the Jinja rendering environment to expose database operations as Jinja-callable methods. Below this, the AdapterImpl provides the concrete implementation of these operations, delegating to specialized metadata adapters for platform-specific introspection and using an underlying AdapterEngine for physical database communication.
Sources: crates/dbt-adapter/src/adapter/mod.rs:100-110, crates/dbt-adapter/src/adapter/adapter_impl.rs:320-350
Core Components
The architecture is composed of several key structures that manage state and execution logic.
The Adapter Facade
The Adapter struct is the primary entry point for Jinja-based method calls. It wraps an InnerAdapter enum, which determines if the adapter is in a "Parse" state (for shallow operations during project parsing) or a "Typed" state (for actual execution).
This diagram illustrates the delegation path from the Jinja environment through the Adapter facade to the underlying execution engines.
Sources: crates/dbt-adapter/src/adapter/mod.rs:88-100, crates/dbt-adapter/src/adapter/mod.rs:114-125
Adapter Implementation (AdapterImpl)
The AdapterImpl struct contains the logic for specific database operations. It handles:
- Connection Management: Borrowing thread-local connections through a
ConnectionGuard. - Statement Splitting: Breaking complex SQL blocks into individual statements based on the adapter's dialect.
- Result Processing: Converting database result sets (Arrow RecordBatches) into
AgateTableobjects for Jinja consumption.
Sources: crates/dbt-adapter/src/adapter/adapter_impl.rs:592-660, crates/dbt-adapter/src/adapter/adapter_impl.rs:2415-2430
SQL Execution Flow
Execution follows a synchronous path within the Jinja thread but utilizes asynchronous capabilities for metadata operations.
Execution Sequence
- Jinja Dispatch: A macro calls a method like
adapter.execute(). - Context Retrieval: The adapter extracts
QueryCtxandCancellationTokenfrom the state. - Connection Acquisition: A thread-local connection is borrowed via the engine.
- Statement Processing: For non-multi-statement warehouses (like Snowflake or Redshift), the SQL is split into individual statements.
- Physical Execution: The
AdapterEngine(often anAdbcEngine) sends the query to the warehouse. - Response Formatting: The engine returns an
AdapterResponsecontaining the row count and query ID.
This sequence shows how a SQL execution request flows from Jinja to the physical driver and back.
Sources: crates/dbt-adapter/src/adapter/mod.rs:1100-1130, crates/dbt-adapter/src/adapter/adapter_impl.rs:592-660
Metadata and Introspection
Introspection logic is encapsulated in the MetadataAdapter trait. Each platform provides a concrete implementation (e.g., SnowflakeMetadataAdapter, DatabricksMetadataAdapter).
Schema Hydration
The adapter can "hydrate" its local RelationCache by listing relations in parallel. This is critical for performance, as it avoids repeated get_relation calls to the warehouse.
| Feature | Snowflake Implementation | Databricks Implementation | DuckDB Implementation |
|---|---|---|---|
| Relation Listing | Uses SHOW OBJECTS with pagination. | Queries system.information_schema.tables. | Queries information_schema.tables. |
| Parallelism | Parallelized across connections via MapReduce. | Parallelized per schema via MapReduce. | Restricted to MAX_CONNECTIONS (4). |
| Freshness | Queries INFORMATION_SCHEMA.TABLES. | Tries DESCRIBE HISTORY (Delta) then IS. | Targeted DESCRIBE probes. |
Sources: crates/dbt-adapter/src/metadata/snowflake/mod.rs:240-270, crates/dbt-adapter/src/metadata/databricks/mod.rs:46-110, crates/dbt-adapter/src/metadata/duckdb/mod.rs:250-300
Freshness Overrides
The architecture supports per-source loaded_at_field or loaded_at_query overrides. When provided, the adapter partitions relations into "Bulk" tasks (standard info-schema queries) and "Override" tasks (custom SQL queries), running them in parallel.
Sources: crates/dbt-adapter/src/metadata/snowflake/mod.rs:434-480, crates/dbt-adapter/src/metadata/bigquery/mod.rs:320-350
Warehouse Capability Mapping
The AdapterImpl maps generic dbt concepts to platform-specific capabilities through several tables and lookup functions.
Incremental Strategies
Different adapters support various subsets of dbt's incremental strategies.
| Adapter Type | Supported Strategies |
|---|---|
| Postgres / DuckDB | Append, DeleteInsert, Merge, Microbatch |
| Snowflake | Append, DeleteInsert, InsertOverwrite, Merge, Microbatch |
| BigQuery | Append |
| Databricks | Append, Merge, InsertOverwrite, ReplaceWhere |
| ClickHouse | Append, DeleteInsert, InsertOverwrite, Microbatch, Legacy |
Sources: crates/dbt-adapter/src/adapter/adapter_impl.rs:480-498
Behavior Flags
Behavior flags control engine-level logic and backward compatibility for specific adapters.
| Flag Name | Adapter | Purpose |
|---|---|---|
use_materialization_v2 | Databricks | Separates create and insert for better performance. |
snowflake_default_transient_dynamic_tables | Snowflake | Dynamic tables default to transient. |
redshift_grants_extended | Redshift | Enables identity-prefixed grantees (user:, role:, group:). |
use_catalogs_v2 | All | Enables experimental catalogs.yml v2 validation. |
Sources: crates/dbt-adapter/src/adapter/adapter_impl.rs:2296-2360
Replay Architecture
For recorded trace playback (conformance testing or dry-runs), the architecture includes a Replayer trait. When the AdapterImpl is initialized in "Replay" mode, it dispatches calls to a replayer that consumes recorded events instead of communicating with a live warehouse.
Summary
The Adapter Architecture provides a high-performance, Rust-native bridge for dbt Fusion. By separating Jinja-facing facades from concrete execution logic and metadata introspection, the system achieves significant parallelism during schema hydration and freshness checks while maintaining a consistent programming model for macro developers. The use of traits and the InnerAdapter pattern allows the system to seamlessly switch between live execution, parse-time shallow analysis, and recorded trace replay.
Sources: crates/dbt-adapter/src/adapter/mod.rs:114-150, crates/dbt-adapter/src/adapter/adapter_impl.rs:320-380