Executing Models (Run & Build)
Relevant source files
The following files were used as context for generating this wiki page:
Executing models within the meltano-dbt-ext environment involves wrapping standard dbt commands such as run and build inside a Meltano-aware extension. This extension manages environment-specific configurations, provides automated pre-invocation cleanup, and ensures dependencies are resolved before execution.
The primary mechanism for execution is the dbt_invoker, which acts as a pass-through shim for the dbt CLI. This ensures that when a user triggers a model execution, the extension handles the necessary setup steps—like cleaning the target directory and installing dbt packages—automatically unless explicitly disabled.
Sources: README.md:1-10, dbt_ext/extension.py:34-40, dbt_ext/pass_through.py:11-18
Command Architecture and Lifecycle
Model execution follows a structured lifecycle managed by the dbt class. Every command passed to the extension goes through a pre_invoke phase before the actual dbt command is executed.
Pre-Invoke Logic
The pre_invoke hook is designed to ensure a clean state for the dbt project. By default, it executes dbt clean followed by dbt deps. This behavior can be toggled using the DBT_EXT_SKIP_PRE_INVOKE environment variable or bypassed automatically if the command being called is itself deps or clean.
Sources: dbt_ext/extension.py:53-83, tests/test_dbt_ext.py:22-43
The Invocation Flow
The following diagram illustrates the flow from a user command (like meltano invoke dbt-postgres:run) through the extension's logic to the final dbt execution.
The flow ensures that dependencies are up-to-date before running models.
Sources: dbt_ext/extension.py:53-100, dbt_ext/pass_through.py:11-18
Execution Commands
The extension supports standard dbt commands for model execution and lifecycle management. These are typically defined in the Meltano plugin definition and passed through to the underlying dbt binary.
| Command | dbt Argument | Description |
|---|---|---|
run | run | Compiles SQL and executes it against the target database. |
build | build | Runs models, tests, snapshots, and seeds in DAG order. |
compile | compile | Generates executable SQL without running it. |
seed | seed | Loads data from CSV files into the data warehouse. |
snapshot | snapshot | Executes snapshots defined in the project. |
Sources: README.md:92-115, dbt_ext/extension.py:85-100
Configuration and Environment Management
Execution behavior is controlled by environment variables and Meltano settings. These variables determine where the project files are located and how the invoker should behave.
Key Configuration Settings
DBT_EXT_TYPE: Required. Specifies the dbt profile type (e.g.,postgres,snowflake).DBT_PROJECT_DIR: The directory where the dbt project resides (defaults totransform).DBT_EXT_SKIP_PRE_INVOKE: If set totrue, the extension will not runcleanordepsbefore model execution.
Sources: dbt_ext/extension.py:34-48, README.md:15-25
Directory Structure
The extension expects a specific structure for execution, which is initialized during the initialize command.
Implementation Details
The execution is handled by the dbt_invoker (mapped to dbt_ext.pass_through:pass_through_cli). This uses the meltano.edk.process.Invoker class to run subprocesses and capture logs.
# From dbt_ext/extension.py
def invoke(self, command_name: str | None, *command_args: ExecArg) -> None:
try:
command_msg = command_name if command_name else self.dbt_bin
if len(command_args) > 0:
command_msg += f" {command_args[0]}"
log.info(f"Extension executing `{command_msg}`...")
self.dbt_invoker.run_and_log(command_name, *command_args)
except subprocess.CalledProcessError as err:
log_subprocess_error(f"dbt {command_name}", err, "dbt invocation failed")
sys.exit(err.returncode)
Sources: dbt_ext/extension.py:85-100, pyproject.toml:23-24
Summary
Executing models in meltano-dbt-ext is a managed process that prioritizes environment consistency. By wrapping the dbt CLI, the extension automates the cleanup and dependency installation steps required for reliable runs in CI/CD and production environments. Users can interact with dbt normally through meltano invoke, while the extension handles the underlying project directory and profile logic.