Data Testing & Snapshots
Relevant source files
The following files were used as context for generating this wiki page:
Data Testing and Snapshots in the meltano-dbt-ext project are managed as specialized commands within the dbt utility extension. This system allows users to validate data integrity and capture point-in-time states of data models directly through the Meltano ecosystem. The extension wraps the standard dbt CLI, providing a structured way to execute these operations against various supported data warehouses like PostgreSQL, Snowflake, and BigQuery.
The integration ensures that testing and snapshotting are part of the standard data lifecycle by automatically handling environment-specific configurations and pre-invocation requirements. This includes managing profile directories, project directories, and ensuring dependencies are resolved before execution.
Sources: README.md:120-143, dbt_ext/extension.py:34-45
Command Execution and Workflow
The extension exposes specific commands for data validation and state capture. These commands are mapped to dbt's native functionality but are invoked through the Meltano extension's dbt_invoker or dbt_extension executables.
Data Testing
The test command runs validations defined within the dbt project. It targets deployed models to ensure they meet schema constraints and business logic requirements.
Snapshots
The snapshot command executes logic to record changes in tables over time, facilitating Type 2 Slowly Changing Dimensions (SCD). This is driven by configurations in the snapshot-paths defined in the project structure.
Unified Execution
The build command provides a consolidated workflow, running models, tests, snapshots, and seeds in the correct Directed Acyclic Graph (DAG) order to maintain data consistency across the pipeline.
Sources: README.md:132-137, dbt_ext/dbt_files/bundle/transform/dbt_project.yml:21-22
The following diagram illustrates the command flow when a user initiates a test or snapshot through the extension:
The diagram shows the lifecycle of an invocation, including the automated execution of clean and deps before the primary command runs.
Sources: dbt_ext/extension.py:53-102, tests/test_dbt_ext.py:21-43
Architecture and Configuration
The extension uses a set of configuration paths and environment variables to locate the necessary dbt files for testing and snapshotting.
File Structure
The project structure is initialized with default paths for models, tests, and snapshots to ensure the dbt engine knows where to look for source code and validation logic.
| Path Type | Default Value | Description |
|---|---|---|
test-paths | tests | Directory containing dbt data tests and schema tests. |
snapshot-paths | snapshots | Directory containing dbt snapshot definitions. |
target-path | target | Output directory for compiled SQL and run results. |
Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:15-23
Pre-Invocation Hooks
To ensure reliability, the extension implements a pre_invoke hook in the dbt class. Unless DBT_EXT_SKIP_PRE_INVOKE is set to true, every call to test, snapshot, or build will first trigger dbt clean and dbt deps.
def pre_invoke(self, invoke_name: str | None, *invoke_args: ExecArg) -> None:
if self.skip_pre_invoke:
return
if invoke_name in ["deps", "clean"]:
return
self.dbt_invoker.run_and_log("clean")
self.dbt_invoker.run_and_log("deps")
Sources: dbt_ext/extension.py:53-85, tests/test_dbt_ext.py:21-43
Environment Management
Testing and snapshots are context-aware, using Meltano environment variables to determine the target database and schema. This allows for isolated testing environments (e.g., dev, staging, prod).
The sequence shows how environment variables flow from Meltano through the extension into the dbt profiles to target the correct data warehouse environment.
Sources: dbt_ext/dbt_files/profiles/postgres/profiles.yml:3-10, dbt_ext/extension.py:34-45
Connection Settings
The extension supports multiple warehouses. Configuration is typically mapped from Meltano config blocks to profiles.yml via environment variables.
| Variable | Description | Source File |
|---|---|---|
MELTANO_ENVIRONMENT | Determines the dbt target (dev, staging, prod). | dbt_ext/dbt_files/profiles/postgres/profiles.yml:3 |
DBT_PROFILES_DIR | Location of the profiles.yml file. | dbt_ext/extension.py:42 |
DBT_PROJECT_DIR | Root of the dbt project (defaults to transform). | dbt_ext/extension.py:39 |
Sources: dbt_ext/extension.py:34-45, README.md:23-35
Summary
The Data Testing & Snapshots system in meltano-dbt-ext provides a robust wrapper around dbt's native capabilities. By automating the setup of environments, managing dependencies through pre-invocation hooks, and mapping Meltano configuration to dbt profiles, it ensures that data quality checks and historical state tracking are executed consistently across different deployment stages. This integration allows developers to treat data tests and snapshots as first-class citizens in their Meltano-managed data pipelines.