Project Configuration Bundling
Relevant source files
The following files were used as context for generating this wiki page:
Project Configuration Bundling in the meltano-dbt-ext utility is the process of deploying a standardized dbt project structure and environment-specific database profiles into a Meltano project. This mechanism ensures that when a user initializes the dbt extension, they receive a functional dbt environment pre-configured to work with Meltano's environment variables and project structure.
The bundling system relies on internal resources stored within the extension's package, which are programmatically copied to the local filesystem during the initialization phase. This includes base project metadata, directory structures for dbt assets (models, macros, seeds), and a profiles.yml tailored to the specific database type defined by the user.
Sources: dbt_ext/extension.py:126-160, README.md:12-25
Initialization Logic and File Deployment
The core of the bundling logic resides in the initialize method of the dbt class. It manages the creation of directories and the recursive copying of bundled files from the extension's internal dbt_files module to the target project directory.
Directory Structure Creation
The extension first verifies the existence of the dbt project directory (defaulting to transform) and the profiles directory. If these do not exist, the extension creates them using Path.mkdir(parents=True).
Sources: dbt_ext/extension.py:128-132, dbt_ext/extension.py:148-150
Transform Bundle Deployment
The extension uses importlib.resources to access the bundle/transform directory within the package. It iterates through all entries (files and directories), excluding __pycache__, and copies them to the dbt_project_dir. Files are copied using shutil.copy, while directories are copied using shutil.copytree with dirs_exist_ok=True.
Sources: dbt_ext/extension.py:134-146
Profile Deployment
Profile bundling is conditional based on the DBT_EXT_TYPE environment variable. The extension searches the profiles resource directory for a folder matching the specified type (e.g., postgres, snowflake). If a match is found, the contents of that directory are deployed to the dbt_profiles_dir.
Sources: dbt_ext/extension.py:152-160
The following diagram illustrates the file deployment flow during initialization:
The diagram shows the sequential steps taken by the initialize command to set up the dbt environment.
Sources: dbt_ext/extension.py:126-168
Bundled Configuration Components
The bundled files provide a skeleton for a dbt project that adheres to Meltano's expectations.
Base dbt Project Configuration
The bundled dbt_project.yml sets up standard paths and disables anonymous usage stats. It specifically defines the profile name as meltano to match the bundled profiles.
| Key | Value | Description |
|---|---|---|
name | my_meltano_project | Default project name |
profile | meltano | Profile name used in profiles.yml |
model-paths | ['models'] | Directory for dbt models |
clean-targets | ['target', 'dbt_packages', 'logs'] | Files removed during dbt clean |
Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:1-25
Profile Template Bundling
Profiles are bundled with Jinja2 templates that reference environment variables. This allows Meltano to inject credentials and configuration dynamically. Profiles typically include dev, staging, and prod targets, with the default target determined by the MELTANO_ENVIRONMENT variable.
meltano:
target: "{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}"
outputs:
dev:
type: postgres
host: "{{ env_var('DBT_POSTGRES_HOST') }}"
# ... other config
Sources: dbt_ext/dbt_files/profiles/postgres/profiles.yml:3-10, dbt_ext/dbt_files/profiles/snowflake/profiles.yml:6-12
Configuration Parameters
The bundling and execution behavior is controlled via environment variables and Meltano settings.
| Setting Name | Environment Variable | Default | Description |
|---|---|---|---|
type | DBT_EXT_TYPE | None (Required) | The dbt adapter type (e.g., snowflake, postgres). |
project_dir | DBT_PROJECT_DIR | transform | The root directory of the dbt project. |
profiles_dir | DBT_PROFILES_DIR | transform/profiles | Where dbt looks for profiles.yml. |
skip_pre_invoke | DBT_EXT_SKIP_PRE_INVOKE | false | If true, skips dbt clean and dbt deps before commands. |
Sources: dbt_ext/extension.py:44-55, README.md:26-38
Summary of Bundling Significance
Project Configuration Bundling enables a "zero-config" start for users integrating dbt into a Meltano project. By automating the deployment of dbt_project.yml and environment-aware profiles.yml files, the extension ensures that the dbt invoker has all necessary context to run commands like run, test, or compile immediately after installation. This structure enforces consistency across different database adapters by centralizing the project skeleton within the extension code.