Quickstart Guide
Relevant source files
The following files were used as context for generating this wiki page:
The meltano-dbt-ext is a Meltano utility extension designed to manage dbt projects within a Meltano environment. It wraps the dbt CLI using the Meltano Extension Developer Kit (EDK), providing standardized commands for initialization, invocation, and configuration management. This extension simplifies the integration of dbt by handling pre-invoke hooks like dbt clean and dbt deps automatically, and by providing pre-configured profile templates for various data warehouses.
Sources: README.md:1-5, dbt_ext/extension.py:28-40, pyproject.toml:5-15
Installation and Basic Usage
To begin using the extension, it must be added to a Meltano project as a utility. The installation involves defining the plugin in the meltano.yml file and then using the Meltano CLI to install and initialize the environment.
Command Sequence
The standard workflow for setting up a dbt extension (using PostgreSQL as an example) is:
- Install:
meltano install utility dbt-postgres - Initialize:
meltano invoke dbt-postgres:initialize - Verify:
meltano invoke dbt-postgres list - Test:
meltano invoke dbt-postgres test
Sources: README.md:213-218
Key CLI Entrypoints
The extension provides two primary scripts for interaction:
| Script Name | Function | Source |
|---|---|---|
dbt_extension | Used for extension-specific commands like initialize and describe. | pyproject.toml:23, dbt_ext/main.py:34-45 |
dbt_invoker | A passthrough shim used to execute standard dbt commands (run, test, etc.). | pyproject.toml:24, dbt_ext/pass_through.py:13-18 |
Core Architecture
The extension is built on the ExtensionBase class from the Meltano EDK. It manages the lifecycle of a dbt project by controlling the execution environment and directory structure.
The diagram shows how Meltano invocations are routed through the extension logic, which handles hooks before executing the target dbt command.
Sources: dbt_ext/extension.py:42-120, dbt_ext/main.py:48-61
Directory Management
During initialization, the extension deploys a default project structure to the path specified by the project_dir setting (defaulting to $MELTANO_PROJECT_ROOT/transform).
- Project Files: Copies the contents of
dbt_files/bundle/transform(includingdbt_project.yml) to the target directory. - Profile Files: Deploys a warehouse-specific
profiles.ymlbased on theDBT_EXT_TYPEsetting.
Sources: dbt_ext/extension.py:144-177, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:1-32
Configuration and Environment Variables
The extension relies on environment variables to determine its behavior and to populate dbt profiles.
Primary Extension Settings
| Setting | Environment Variable | Default | Description |
|---|---|---|---|
type | DBT_EXT_TYPE | Required | Specifies the dbt profile to use (e.g., postgres, snowflake). |
skip_pre_invoke | DBT_EXT_SKIP_PRE_INVOKE | false | If true, skips the automatic execution of dbt clean and dbt deps. |
project_dir | DBT_PROJECT_DIR | transform | The directory where the dbt project resides. |
profiles_dir | DBT_PROFILES_DIR | transform/profiles | The directory where profiles.yml is located. |
Sources: dbt_ext/extension.py:43-55, README.md:14-30
Execution Logic: Pre-Invoke Hooks
Whenever a dbt command (other than deps or clean) is invoked, the extension automatically performs a cleanup and dependency refresh unless explicitly skipped.
The sequence diagram illustrates the automated execution of clean and deps prior to the user-requested command.
Sources: dbt_ext/extension.py:57-93, tests/test_dbt_ext.py:20-45
Profiles and Targets
The extension includes templates for several data warehouses. These templates use Jinja2 to pull credentials from environment variables, typically prefixed with DBT_<TYPE>_.
Supported Warehouses
The extension provides bundled profiles for:
- PostgreSQL
- Snowflake
- Athena
- Redshift
- BigQuery
- ClickHouse
- DuckDB
Sources: dbt_ext/dbt_files/profiles/
Profile Structure
All bundled profiles use meltano as the profile name and default to MELTANO_ENVIRONMENT (or dev) as the target. They typically define three outputs: dev, staging, and prod.
meltano:
target: "{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}"
outputs:
dev:
type: postgres
host: "{{ env_var('DBT_POSTGRES_HOST') }}"
# ... other settings
Sources: dbt_ext/dbt_files/profiles/postgres/profiles.yml:3-15, dbt_ext/dbt_files/profiles/snowflake/profiles.yml:8-20
Conclusion
The meltano-dbt-ext provides a robust wrapper for dbt, ensuring that dbt projects in Meltano are consistently initialized and executed. By automating dependency management and providing environment-aware profile templates, it reduces the boilerplate required to maintain data transformation pipelines. Developers should ensure DBT_EXT_TYPE is correctly set to match their target warehouse to enable proper profile deployment during initialization.
Sources: dbt_ext/extension.py:180-185, README.md:7-12