BigQuery Adapter Integration

Relevant source files

The following files were used as context for generating this wiki page:

• dbt_ext/dbt_files/profiles/bigquery/profiles.yml
• dbt_ext/extension.py
• dbt_ext/main.py
• README.md
• dbt_ext/dbt_files/bundle/transform/dbt_project.yml

Introduction

The BigQuery Adapter Integration within the meltano-dbt-ext project provides a structured mechanism to manage dbt transformations on Google BigQuery through Meltano. This integration utilizes a custom extension framework to wrap the dbt CLI, facilitating automated environment configuration and profile management.

The system relies on environment variables to dynamically populate dbt profiles, supporting multiple deployment environments such as development, staging, and production. It handles the deployment of necessary configuration files and ensures that dbt commands are executed within a pre-configured environment consistent with Meltano's lifecycle.

Sources: README.md:1-24, dbt_ext/extension.py:34-51, dbt_ext/dbt_files/profiles/bigquery/profiles.yml:1-36

Architecture and Initialization

The integration is built upon the dbt class, which inherits from ExtensionBase. During initialization, the extension verifies the DBT_EXT_TYPE environment variable, which must be set to bigquery for this specific adapter.

Initialization Workflow

When the initialize command is invoked, the extension performs several file-system operations to set up the dbt environment:

1. Directory Creation: It ensures the DBT_PROJECT_DIR (defaulting to transform) exists.
2. Bundle Deployment: It copies core dbt project files (like dbt_project.yml) from the internal bundle to the project directory.
3. Profile Deployment: It identifies the BigQuery profile directory and copies the profiles.yml to the targeted profiles directory.

The flow above illustrates how the extension prepares the local filesystem for dbt execution by mapping internal resources to the user's project structure.
Sources: dbt_ext/extension.py:34-58, dbt_ext/extension.py:126-155, dbt_ext/main.py:31-43

BigQuery Profile Configuration

The profiles.yml for BigQuery is designed to be fully dynamic, relying on Meltano's environment variables. This allows users to switch between authentication methods and environments without modifying the YAML file directly.

Environment Targets

The profile defines three distinct targets based on the MELTANO_ENVIRONMENT variable:

• dev: Configured with 2 threads.
• staging: Configured with 4 threads.
• prod: Configured with 6 threads.

Configuration Parameters

The following table details the mapping of environment variables to BigQuery profile settings:

Profile Field	Environment Variable	Description
method	DBT_BIGQUERY_AUTH_METHOD	The authentication method (e.g., oauth, service-account).
project	DBT_BIGQUERY_PROJECT	The Google Cloud project ID.
dataset	DBT_BIGQUERY_DATASET	The default dataset for dbt models.
keyfile	DBT_BIGQUERY_KEYFILE	Path to the service account JSON key file.
refresh_token	DBT_BIGQUERY_REFRESH_TOKEN	OAuth2 refresh token.
client_id	DBT_BIGQUERY_CLIENT_ID	OAuth2 client ID.
client_secret	DBT_BIGQUERY_CLIENT_SECRET	OAuth2 client secret.

Sources: dbt_ext/dbt_files/profiles/bigquery/profiles.yml:4-36, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:3-4

Execution Logic

Commands are executed through the dbt_invoker (defined in pass_through.py) or the dbt_extension entry point. The extension implements a pre_invoke hook that automatically runs maintenance commands unless explicitly skipped.

Pre-invoke Hook

Before running standard dbt commands (like run or test), the extension executes:

1. dbt clean: To remove old artifacts.
2. dbt deps: To ensure all dbt packages are installed.

This behavior is bypassed if the command being executed is itself clean or deps, or if DBT_EXT_SKIP_PRE_INVOKE is set to true.

The sequence diagram demonstrates the automated dependency management provided by the extension wrapper.
Sources: dbt_ext/extension.py:60-93, dbt_ext/pass_through.py:14-20, dbt_ext/main.py:46-58

Project Structure Integration

The integration assumes a specific directory layout defined in dbt_project.yml. This structure ensures that dbt looks for models and seeds in locations consistent with the files deployed during initialization.

# dbt_ext/dbt_files/bundle/transform/dbt_project.yml snippets
model-paths: ["models"]
seed-paths: ["data"]
target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"
  - "logs"

Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:10-25

Conclusion

The BigQuery Adapter Integration streamlines dbt operations within a Meltano environment by abstracting profile management and automating setup tasks. By utilizing the dbt_invoker and dynamic environment variables, it provides a consistent interface for developers to build and test transformations on BigQuery across multiple deployment tiers.