dbt-bigquery Adapter
Relevant source files
The following files were used as context for generating this wiki page:
The dbt-bigquery adapter is a specialized plugin that enables dbt (data build tool) to interface with Google BigQuery. It facilitates the "Transform" step in ELT workflows by allowing data analysts and engineers to manage BigQuery resources using software engineering best practices such as version control and modularity.
Unlike many SQL-based adapters in the monorepo, the BigQuery adapter extends BaseAdapter directly rather than SQLAdapter, utilizing the google-cloud-bigquery library for execution. It supports BigQuery-specific features such as time partitioning, clustering, and specialized data types like STRUCT and ARRAY.
Sources: dbt-bigquery/README.md:26-30, CLAUDE.md:144-147
Architecture and Components
The adapter is structured as an independent Python package within the dbt-adapters monorepo. It interacts with the BigQuery API to manage connections, execute queries, and maintain relation metadata.
Core Structure
The implementation is organized into several key modules:
connections.py: Manages BigQueryConnectionManagerandCredentials.impl.py: Contains the primaryBigQueryAdapterimplementation.relation.py: Defines BigQuery-specific relation handling.column.py: Handles complex BigQuery types includingSTRUCT,ARRAY,GEOGRAPHY, andJSON.python_submissions.py: Manages the execution of Python models via BigFrames or Dataproc.
Sources: CLAUDE.md:105-117, CLAUDE.md:144-150, AGENTS.md:11-20
Connection Lifecycle
The following diagram illustrates how the adapter establishes a connection and executes a job in BigQuery.
The adapter manages connection pooling and thread-safe execution, wrapping query execution with retry logic for transient errors like 503s or rate limits.
Sources: CLAUDE.md:174-180, dbt-bigquery/CHANGELOG.md:111-114, dbt-adapters/README.md:31-35
Key Features and Configuration
The adapter supports a wide array of BigQuery-specific configurations for performance and cost management.
Materialization Configurations
| Configuration | Description |
|---|---|
partition_by | Configures time or integer range partitioning to optimize costs. |
cluster_by | Defines columns for data clustering to improve query performance. |
maximum_bytes_billed | Limits the cost of a single query by failing if it exceeds the limit. |
labels | Applies key-value pairs to BigQuery resources for cost tracking. |
priority | Sets job priority to batch or interactive. |
copy_partitions | Enables incremental strategy for insert_overwrite via partition copying. |
Sources: CLAUDE.md:154-172, dbt-bigquery/CHANGELOG.md:46-48
Partitioning Logic
BigQuery supports time-based and integer-range partitioning. The adapter handles the rendering of these configurations into BigQuery DDL.
Python Models and BigFrames
The BigQuery adapter supports Python models, which can be executed using Google's BigFrames library. This allows users to write DataFrame-style code that is pushed down to the BigQuery engine.
- Authentication: Supports Service Account keys, OAuth, Application Default Credentials (ADC), and Impersonation.
- Execution: Can utilize custom runtime templates and specific Dataproc regions for notebook execution.
- Configuration: Includes settings for
compute_region,job_timeout_ms, andresource_tags.
Sources: CLAUDE.md:174-178, dbt-bigquery/CHANGELOG.md:144-150, dbt-bigquery/CHANGELOG.md:197-200
External Catalog Integration
BigQuery supports external catalog integrations, specifically for BigLake metastore and Apache Iceberg tables.
- BigLake Integration: Facilitates cross-cloud or external data access via the
biglakecatalog type. - Iceberg Support: Supports
table_format='iceberg'in materialization configs, enabling BigQuery to manage Iceberg tables on Cloud Storage. - V2 Catalogs: Implements
Capability.CatalogsV2to provide enhanced metadata handling for these external formats.
Sources: CLAUDE.md:209-215, dbt-bigquery/CHANGELOG.md:175-177, dbt-bigquery/CHANGELOG.md:95-97
Summary
The dbt-bigquery adapter is a robust implementation designed to leverage BigQuery's unique architecture. By bypassing the standard SQLAdapter in favor of a direct BaseAdapter implementation, it provides deep integration for Google Cloud features, including complex data types, cost controls through partitioning/clustering, and modern data processing via Python and BigFrames.