Installation Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• pyproject.toml
• dbt_ext/extension.py
• dbt_ext/main.py
• dbt_ext/dbt_files/bundle/transform/dbt_project.yml
• dbt_ext/pass_through.py

The meltano-dbt-ext is a Meltano utility extension designed to manage dbt projects. It wraps the dbt CLI, providing enhanced integration with the Meltano ecosystem by automating project initialization and managing database-specific profiles through environment variables. The extension supports various dbt adapters including Postgres, Snowflake, Redshift, Athena, Clickhouse, BigQuery, and DuckDB.

Sources: pyproject.toml:1-15, dbt_ext/extension.py:27-30, README.md:1-10

Prerequisites

Before installing the extension, ensure the following requirements are met:

• Python: Version 3.10 or higher is required.
• Meltano: A working Meltano project.
• Dependencies: The extension relies on PyYAML, typer, and meltano.edk (version 0.5 to 0.7).

Sources: pyproject.toml:9-15

Core Installation Steps

The installation process involves adding the specific dbt adapter utility to a Meltano project and initializing the extension to deploy required files and profiles.

1. Adding the Plugin

Users install the extension as a utility plugin using the Meltano CLI. The specific plugin name corresponds to the desired dbt adapter (e.g., dbt-postgres).

meltano install utility dbt-postgres

Sources: README.md:195-196

2. Initialization

After installation, the initialize command must be invoked. This command triggers the dbt.initialize() method, which performs several critical setup tasks:

• Creates the dbt project directory (default: transform/).
• Deploys the standard dbt_project.yml and project structure.
• Deploys the specific profiles.yml corresponding to the DBT_EXT_TYPE setting.

meltano invoke dbt-postgres:initialize

Sources: dbt_ext/extension.py:126-163, README.md:197

Installation and Initialization Flow

The following diagram illustrates the sequence of operations when a user installs and initializes the extension.

The initialization logic ensures that the DBT_EXT_TYPE environment variable matches a directory in the dbt_files/profiles internal resource folder.
Sources: dbt_ext/extension.py:126-163, dbt_ext/main.py:34-46

Configuration

The extension requires specific environment variables and plugin settings to function correctly. These are typically defined in the meltano.yml file.

Required Settings

Setting	Environment Variable	Default	Description
type	DBT_EXT_TYPE	N/A	The dbt profile type (e.g., postgres, snowflake). Required.
project_dir	DBT_PROJECT_DIR	transform	Path to the dbt project directory.
profiles_dir	DBT_PROFILES_DIR	transform/profiles	Path to the directory containing profiles.yml.
skip_pre_invoke	DBT_EXT_SKIP_PRE_INVOKE	false	If true, skips automatic dbt clean and dbt deps.

Sources: dbt_ext/extension.py:41-55, README.md:14-30

Executable Selection

The extension defines two primary entry points in pyproject.toml:

• dbt_extension: Maps to dbt_ext.main:app, used for extension-specific commands like initialize and describe.
• dbt_invoker: Maps to dbt_ext.pass_through:pass_through_cli, used as a shim to pass commands directly to the underlying dbt binary.

Sources: pyproject.toml:22-24, dbt_ext/pass_through.py:12-19

Automated Pre-Invoke Hooks

Upon successful installation and during standard operation, the extension automatically executes pre-invoke hooks to maintain project state, unless DBT_EXT_SKIP_PRE_INVOKE is set to true.

The extension runs dbt clean and dbt deps before execution to ensure dependencies are fresh. These steps are skipped if the primary command is itself clean or deps to avoid infinite loops.
Sources: dbt_ext/extension.py:57-89

Verification

To verify the installation, users can run standard dbt commands through the Meltano invoker:

meltano invoke dbt-postgres list
meltano invoke dbt-postgres test

Sources: README.md:198-199