Adding New dbt Adapters
Relevant source files
The following files were used as context for generating this wiki page:
Adding a new dbt adapter to the meltano-dbt-ext utility requires defining a specific profile configuration and ensuring the extension can correctly initialize the adapter's files. The extension uses the DBT_EXT_TYPE environment variable to determine which specific profile bundle to deploy during the initialization phase.
This process involves creating a new directory structure within the profiles resource path and defining environment-variable-driven profiles.yml files that map Meltano configuration to dbt-specific settings.
Adapter Architecture and Deployment
The extension follows a standardized deployment flow during the initialize command. It copies a base dbt_project.yml and then selectively copies the adapter-specific profile folder based on the user-provided type.
The diagram shows the initialization flow where the extension selects the adapter profile based on environment variables.
Sources: dbt_ext/extension.py:118-154
Key Components for New Adapters
| Component | Responsibility | File Reference |
|---|---|---|
DBT_EXT_TYPE | Env var indicating the adapter name (e.g., postgres, snowflake). | dbt_ext/extension.py:38-41 |
profiles/ Directory | Contains subdirectories named after the DBT_EXT_TYPE. | dbt_ext/extension.py:144-152 |
profiles.yml | Template file using Jinja to map env vars to dbt configs. | dbt_ext/dbt_files/profiles/postgres/profiles.yml:1-12 |
Profile Configuration Requirements
New adapters must provide a profiles.yml located in dbt_ext/dbt_files/profiles/<adapter_name>/. These files must support three standard Meltano environments: dev, staging, and prod. Each environment should map to its respective environment variables using the env_var helper.
Standard Environment Mapping
Every profiles.yml should follow this target structure:
meltano:
target: "{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}"
outputs:
dev:
type: <adapter_type>
# ... adapter specific settings
Sources: dbt_ext/dbt_files/profiles/duckdb/profiles.yml:1-7, dbt_ext/dbt_files/profiles/athena/profiles.yml:9-12
Adapter-Specific Parameter Examples
Different adapters require different connection parameters. When adding a new one, refer to existing implementations for standard patterns:
- DuckDB: Requires a simple
pathvariable. dbt_ext/dbt_files/profiles/duckdb/profiles.yml:8 - Postgres: Requires
host,user,password,port, anddbname. dbt_ext/dbt_files/profiles/postgres/profiles.yml:6-11 - Snowflake: Requires
account,warehouse,role, anddatabase. dbt_ext/dbt_files/profiles/snowflake/profiles.yml:13-20
Implementation Steps
1. Create the Profile Directory
Create a new folder in dbt_ext/dbt_files/profiles/ named exactly what the DBT_EXT_TYPE will be (e.g., my_new_db).
Sources: dbt_ext/extension.py:144-146
2. Define the Profile Template
Create a profiles.yml within that folder. Use environment variables prefixed with DBT_<ADAPTER_NAME>_ to ensure consistency with Meltano's configuration mapping.
# Example of how the extension finds the directory
for entry in files_dir.joinpath("profiles").iterdir():
if entry.name == self.dbt_ext_type and entry.is_dir():
shutil.copytree(entry, self.dbt_profiles_dir, dirs_exist_ok=True)
Sources: dbt_ext/extension.py:144-150
3. Register in Meltano Plugin Definition
Update the README.md or the meltano.yml template to include the new settings required for the adapter. Each entry in the profiles.yml should have a corresponding setting in the plugin definition to allow Meltano to pass the values as environment variables.
Sources: README.md:16-65
Validation
Once the adapter files are added, the extension can be tested by setting the environment variable and running initialize:
export DBT_EXT_TYPE=my_new_db
meltano invoke dbt:initialize
If the directory dbt_ext/dbt_files/profiles/my_new_db does not exist, the extension will log an error: dbt type my_new_db had no matching profile.
Sources: dbt_ext/extension.py:152-154
Summary
Adding a new dbt adapter primarily involves creating a standardized profiles.yml within a specific subdirectory of the extension's resource bundle. This allows the Meltano dbt extension to dynamically deploy the correct configuration based on the DBT_EXT_TYPE setting, enabling seamless integration between Meltano environments and dbt's profile-based connection management.