ADBC Database Drivers
Relevant source files
The following files were used as context for generating this wiki page:
The Arrow Database Connectivity (ADBC) protocol serves as the next-generation foundation for database interactions within dbt Core v2.0. By utilizing ADBC, the project achieves high-performance, Arrow-native data transfers between the Rust-based dbt engine and various database backends.
ADBC drivers are implemented as shared libraries that are loaded dynamically at runtime. This architecture allows dbt Core to remain a self-contained binary while maintaining compatibility with diverse warehouses like Snowflake, BigQuery, and DuckDB.
Sources: CONTRIBUTING.md:12, crates/dbt-adbc/README.md:9-11
Architecture and Components
The ADBC implementation in dbt follows a layered approach where a Rust wrapper provides a type-safe interface to underlying dynamic libraries (drivers).
The diagram shows the hierarchical relationship from the Rust application builders to the physical warehouse through the ADBC shared library.
Sources: crates/dbt-adbc/README.md:16-56, crates/dbt-metricflow/tests/docs_examples.rs:18-35
Key abstractions
The system relies on several core abstractions to manage the lifecycle of a database interaction:
| Component | Description |
|---|---|
| Driver | The loaded shared library responsible for backend-specific protocol implementation. |
| Backend | An enumeration identifying the specific database type (e.g., Snowflake, BigQuery, DuckDBExtended). |
| Database | A configured instance of a driver, often containing connection URIs or credentials. |
| Connection | An active session with the database used to create statements. |
| Statement | The object used to execute SQL queries and retrieve results as Arrow RecordBatch objects. |
Sources: crates/dbt-adbc/src/repl.rs:109-150, crates/dbt-adbc/README.md:16-43
Driver Lifecycle and Management
Loading Strategy
Drivers are typically not bundled directly with the application binary to keep the installation lightweight. Instead, dbt utilizes a dynamic loading strategy.
- System Discovery: The application first looks for drivers in local system paths or specific developer directories (e.g.,
~/code/fs/lib). - CDN Download: If not found locally, drivers are automatically downloaded from the dbt CDN (
public.cdn.getdbt.com) into a local cache directory (/var/cache/dbt). - Dynamic Loading: The shared library is loaded using a
LoadStrategy.
Sources: crates/dbt-adbc/README.md:12-13, crates/dbt-docs-server/README.md:112-116, crates/dbt-adbc/src/repl.rs:135-140
Environment Configuration
Developers can control driver loading behavior through specific environment variables:
| Environment Variable | Purpose |
|---|---|
DISABLE_CDN_DRIVER_CACHE | Prevents the application from using or downloading drivers from the CDN, forcing local usage. |
DISABLE_AUTO_DRIVER_REBUILD | Prevents the system from attempting to rebuild Go-based drivers during development. |
Sources: crates/dbt-adbc/README.md:76-80, crates/dbt-adbc/README.md:104-106
Supported Backends and Driver Implementation
Most ADBC drivers used in the project are written in Go or C, leveraging official SDKs for their respective platforms.
| Backend | Driver Source / Type | Implementation Details |
|---|---|---|
| Snowflake | Go Arrow Snowflake Driver | Uses Go SDKs, loaded as a shared library. |
| BigQuery | Go Arrow BigQuery Driver | Uses Go SDKs, loaded as a shared library. |
| Postgres | C Arrow Postgres Driver | C-based implementation for high performance. |
| DuckDB | Bundled / CMake | Built specifically to include Arrow support and often handled via Backend::DuckDBExtended. |
Sources: CONTRIBUTING.md:13-17, crates/dbt-adbc/README.md:119-125, crates/dbt-metricflow/tests/docs_examples.rs:18-20
Interactive Querying and REPL
The project includes a Read-Eval-Print Loop (REPL) tool that is tightly coupled with ADBC drivers to facilitate rapid testing and feedback.
The sequence shows how the REPL interacts with the driver manager to load libraries and execute queries.
Sources: crates/dbt-adbc/src/repl.rs:260-350, crates/dbt-adbc/README.md:162-166
The REPL supports several commands for inspecting database objects and data:
:get-objects: Lists catalogs, schemas, and tables.:get-schema: Displays the Arrow schema for a specific table.:show-batch: Displays the currentRecordBatchof data extracted from theRecordBatchReader.:move: Navigates between result batches.
Sources: crates/dbt-adbc/src/repl.rs:205-240
Summary
ADBC drivers provide the critical link between dbt Core's Rust engine and the underlying data warehouses. By abstracting the connection through dynamic shared libraries and Arrow-native protocols, dbt achieves high-performance data transfer without a heavy distribution footprint. The lifecycle management—including automatic CDN downloads and local development overrides—ensures both ease of use for end-users and flexibility for contributors.
Sources: crates/dbt-adbc/README.md:9-13, crates/dbt-docs-server/README.md:112-116