Redshift Adapter Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Redshift Adapter Integration within the meltano-dbt-ext project enables Meltano to manage dbt projects specifically targeting Amazon Redshift. It provides a standardized environment-aware configuration through profiles.yml and automates the setup of the dbt execution environment. This integration ensures that dbt commands like run, test, and compile are executed with the correct credentials and connection parameters sourced from Meltano environment variables.
Sources: README.md, dbt_ext/extension.py:27-44, dbt_ext/dbt_files/profiles/redshift/profiles.yml
Architecture and Configuration
The integration relies on the Meltano Extension Developer Kit (EDK) to wrap the dbt CLI. When a user initializes the extension with the Redshift type, the system deploys a specific Redshift profiles.yml and a standard dbt_project.yml.
Initialization Logic
The dbt.initialize() method identifies the required adapter based on the DBT_EXT_TYPE environment variable. If set to redshift, it copies the Redshift-specific profile templates into the project's profiles directory.
Profile Structures
The Redshift profile defines three standard Meltano environments: dev, staging, and prod. All connection details are dynamic, utilizing Jinja2 templates to pull from environment variables.
| Parameter | Default / Type | Environment Variable |
|---|---|---|
method | iam | DBT_REDSHIFT_AUTH_METHOD |
cluster_id | Required | DBT_REDSHIFT_CLUSTER_ID |
iam_duration | 900 (int) | DBT_REDSHIFT_IAM_DURATION_SECONDS |
port | 5439 (int) | DBT_REDSHIFT_PORT |
keepalives_idle | 240 (int) | DBT_REDSHIFT_KEEPALIVES_IDLE |
ra3_node | False | DBT_REDSHIFT_RA3_NODE |
Sources: dbt_ext/dbt_files/profiles/redshift/profiles.yml:6-30
Connection and Authentication
The Redshift adapter supports multiple authentication methods, with a focus on AWS IAM-based authentication.
Authentication Methods
- IAM Authentication: Default method using
cluster_id,iam_profile, andiam_duration_seconds. - Standard Auth: Uses
host,user, andpasswordif specified in environment variables.
Data Flow for Execution
When a command is invoked (e.g., meltano invoke dbt-redshift run), the following sequence occurs:
Pre-Invoke Hooks
By default, the Redshift integration ensures the environment is ready by executing dbt clean and dbt deps before the primary command. This behavior can be toggled using the DBT_EXT_SKIP_PRE_INVOKE setting.
- dbt clean: Removes
target/anddbt_packages/as defined indbt_project.yml. - dbt deps: Installs necessary packages defined in the project.
Sources: dbt_ext/extension.py:58-81, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:25-28
Project Structure Integration
The integration expects a specific directory structure within the Meltano project:
- Project Directory:
$MELTANO_PROJECT_ROOT/transform(Default). - Profiles Directory:
$MELTANO_PROJECT_ROOT/transform/profiles/redshift.
The dbt_project.yml is configured with profile: meltano to match the key in the generated profiles.yml.
Sources: dbt_ext/extension.py:38-42, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:3, README.md
Conclusion
The Redshift Adapter Integration provides a robust, environment-aware wrapper for dbt within Meltano. By leveraging the EDK and standardized template files, it automates Redshift-specific configurations such as IAM authentication and RA3 node support, allowing developers to focus on model development rather than infrastructure plumbing.