Snowflake Adapter Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Snowflake Adapter Integration within meltano-dbt-ext allows Meltano to manage and execute dbt projects targeting Snowflake data warehouses. It functions by wrapping the standard dbt CLI and providing automated configuration injection via environment variables, ensuring that Snowflake-specific connection parameters are correctly mapped from Meltano settings to dbt profiles.
This integration utilizes a dedicated plugin type, dbt-snowflake, which includes the necessary dbt-core and dbt-snowflake dependencies. The system automates the initialization of project structures and the deployment of pre-configured profiles.yml files tailored for Snowflake environments.
Sources: README.md:104-109, dbt_ext/extension.py:27-46
Architecture and Initialization
The integration is built upon the dbt class, which extends ExtensionBase. During the initialization phase, the extension identifies the adapter type through the DBT_EXT_TYPE environment variable. For Snowflake integrations, this is set to snowflake.
Initialization Workflow
When the initialize command is executed, the extension performs the following steps:
- Project Directory Creation: Creates the directory specified by
DBT_PROJECT_DIR(defaulting totransform). - Base Bundle Deployment: Copies core dbt project files from the
bundle/transformdirectory. - Profile Deployment: Locates the
snowflakedirectory within the extension's internalprofilesresource and copies its contents to the targetprofiles_dir.
Sources: dbt_ext/extension.py:114-150, dbt_ext/main.py:32-44
The following diagram illustrates the initialization process for the Snowflake adapter:
The diagram shows how the extension validates the adapter type before deploying the specific configuration files for Snowflake.
Sources: dbt_ext/extension.py:114-154
Configuration and Environment Mapping
The Snowflake integration relies heavily on environment variables to populate the profiles.yml file. The extension uses a unified profile name, meltano, with different targets (dev, staging, prod) determined by the MELTANO_ENVIRONMENT variable.
Snowflake Profile Structure
The profiles.yml for Snowflake defines three standard outputs, each increasing in thread count:
- dev: 2 threads
- staging: 4 threads
- prod: 6 threads
Sources: dbt_ext/dbt_files/profiles/snowflake/profiles.yml:6-38
Key Configuration Parameters
| Setting | Environment Variable | Description |
|---|---|---|
account | DBT_SNOWFLAKE_ACCOUNT | The Snowflake account identifier. |
user | DBT_SNOWFLAKE_USER | The username for authentication. |
password | DBT_SNOWFLAKE_PASSWORD | The user password. |
role | DBT_SNOWFLAKE_ROLE | The Snowflake role to assume. |
warehouse | DBT_SNOWFLAKE_WAREHOUSE | The compute warehouse to use. |
database | DBT_SNOWFLAKE_DATABASE | The target database for models. |
schema | DBT_SNOWFLAKE_SCHEMA | The default schema for models. |
Sources: README.md:124-152, dbt_ext/dbt_files/profiles/snowflake/profiles.yml:11-17
Execution Flow and Pre-invoke Hooks
The extension uses a wrapper called dbt_invoker to execute dbt commands. Before most commands (like run, test, or build), the extension automatically runs "pre-invoke" hooks to ensure the environment is ready.
Sequence of Execution
- Check Skip Flag: If
DBT_EXT_SKIP_PRE_INVOKEis true, hooks are bypassed. - dbt clean: Removes existing artifacts in
target/,dbt_packages/, andlogs/. - dbt deps: Installs required dbt packages.
- Primary Command: Executes the requested dbt command (e.g.,
dbt run).
Sources: dbt_ext/extension.py:53-90, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:25-28
The sequence diagram demonstrates the lifecycle of a command invocation, including the automatic execution of maintenance commands before reaching the Snowflake warehouse.
Sources: dbt_ext/extension.py:53-108, dbt_ext/main.py:48-60
Project Structure Integration
The extension enforces a specific directory structure for the dbt project to ensure compatibility with Meltano's expectations. This is defined in the dbt_project.yml file deployed during initialization.
Default Path Mappings
- Models:
models - Analysis:
analysis - Tests:
tests - Seeds:
data - Packages:
dbt_packages
Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:10-24
Summary
The Snowflake Adapter Integration provides a structured way to run dbt against Snowflake within a Meltano environment. By automating the creation of profiles.yml and mapping Meltano settings to Snowflake-specific environment variables, it reduces manual configuration and ensures consistent execution environments across development, staging, and production tiers.