Command Invocation Flow

Relevant source files

The following files were used as context for generating this wiki page:

• dbt_ext/main.py
• dbt_ext/extension.py
• dbt_ext/pass_through.py
• pyproject.toml
• README.md
• tests/test_dbt_ext.py

The command invocation flow in meltano-dbt-ext defines how user-initiated commands are processed, validated, and executed through the Meltano Extension Developer Kit (EDK). This extension acts as a wrapper around the dbt CLI, providing managed initialization, configuration injection via environment variables, and pre-invocation hooks to maintain project dependencies.

The system supports two primary entry points: the dbt_extension, which manages administrative tasks like project initialization and description, and the dbt_invoker, which serves as a passthrough for standard dbt commands (e.g., run, test, compile).
Sources: pyproject.toml:25-26, README.md:12-14, dbt_ext/extension.py:34-40

Entry Points and CLI Routing

The extension exposes two main scripts defined in the project configuration. These scripts route user input to either the full Typer application or a dedicated passthrough shim.

• dbt_extension: Maps to dbt_ext.main:app. This handles structured commands like initialize and describe.
• dbt_invoker: Maps to dbt_ext.pass_through:pass_through_cli. This is the default executable for most Meltano plugin configurations and provides a direct line to the underlying dbt binary.

Sources: pyproject.toml:25-26, dbt_ext/main.py:21-25, dbt_ext/pass_through.py:12-18

Invocation Sequence Diagram

The following diagram illustrates the flow from a user command to the final execution of the dbt binary.

The flow ensures that the environment is prepared (dependencies installed and targets cleaned) before the primary command executes.
Sources: dbt_ext/extension.py:59-106, dbt_ext/pass_through.py:12-18

Pre-Invoke Logic

A critical feature of the invocation flow is the pre_invoke hook. This logic ensures that the dbt environment is consistent before executing models. By default, the extension runs dbt clean followed by dbt deps.

Hook Execution Rules

The execution of these hooks is conditional based on environment configuration and the specific command being run:

1. Global Skip: If the environment variable DBT_EXT_SKIP_PRE_INVOKE is set to true, all pre-invoke steps are bypassed.
2. Redundancy Check: If the user is explicitly calling deps or clean, the pre-invoke hook skips these steps to prevent circular or redundant executions.
3. Automatic Cleanup: For all other commands (e.g., run, test, build), the extension executes dbt clean and dbt deps sequentially.

Sources: dbt_ext/extension.py:68-91, tests/test_dbt_ext.py:23-45

Extension Initialization Flow

Before commands can be invoked successfully, the project must be initialized. The initialize command sets up the directory structure and deploys the necessary profiles.yml files based on the configured DBT_EXT_TYPE.

Configuration Summary

The behavior of the invocation flow is controlled by several key settings and environment variables.

Setting	Environment Variable	Default	Description
type	DBT_EXT_TYPE	Required	Determines which profiles.yml to deploy (e.g., postgres, snowflake).
project_dir	DBT_PROJECT_DIR	transform	The working directory where dbt commands are executed.
profiles_dir	DBT_PROFILES_DIR	transform/profiles	The directory containing profiles.yml.
skip_pre_invoke	DBT_EXT_SKIP_PRE_INVOKE	false	When true, disables the automatic clean and deps execution.

Sources: dbt_ext/extension.py:44-57, README.md:20-33

Execution Environment Logic

The extension uses the meltano.edk.process.Invoker to manage the subprocess lifecycle. When a command is invoked, the extension:

1. Resolves the current working directory to DBT_PROJECT_DIR.
2. Logs the command being executed to the configured output.
3. Captures subprocess.CalledProcessError to provide formatted error logging via log_subprocess_error.
4. Exits with the return code of the underlying dbt process to ensure Meltano pipelines correctly detect failures.

Sources: dbt_ext/extension.py:53, 98-106

The command invocation flow effectively bridges the gap between Meltano's plugin architecture and dbt's requirement for specific directory structures and dependency management, ensuring a "just works" experience for data engineers.