Seeding & Source Freshness

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• dbt_ext/extension.py
• dbt_ext/dbt_files/bundle/transform/dbt_project.yml
• dbt_ext/main.py
• tests/test_dbt_ext.py

Seeding and Source Freshness are core dbt commands exposed by the meltano-dbt-ext extension to manage data warehouse lifecycles within a Meltano project. These features allow developers to load static data from CSV files and validate the latency of incoming source data, ensuring the integrity and timeliness of the transformation layer.

The extension wraps these standard dbt operations through a passthrough invoker, enabling them to be executed as Meltano commands. This integration ensures that environment variables, project directories, and database profiles are correctly mapped between Meltano's configuration and the underlying dbt execution environment.

Sources: README.md:129-132, README.md:204-207, dbt_ext/extension.py:100-116

Command Implementation and Workflow

The extension implements these features by mapping Meltano commands to specific dbt arguments. When a user invokes seed or freshness, the extension's dbt_invoker translates these into dbt seed and dbt source freshness respectively.

Execution Lifecycle

Before the actual command is executed, the extension performs a pre_invoke step. By default, this step runs dbt clean and dbt deps to ensure the environment is fresh and all packages are installed. This behavior can be toggled using the DBT_EXT_SKIP_PRE_INVOKE setting.

The diagram shows the command execution flow, highlighting the optional pre-invoke automation that prepares the environment for seeding or freshness checks.
Sources: dbt_ext/extension.py:58-98, tests/test_dbt_ext.py:22-42

Data Seeding

Seeding allows users to load small, static datasets (typically CSV files) into the data warehouse. In the meltano-dbt-ext configuration, the seed command is explicitly defined to facilitate this process.

Seed Configuration

The project defines specific paths for seed data within the standard dbt project structure. The default location for seed files is the data directory relative to the project root.

Feature	Configuration Path	Description
Seed Paths	seed-paths: ["data"]	Defines where dbt looks for CSV files to load.
Clean Targets	clean-targets: ["target", ...]	Directories removed during dbt clean before seeding.
Command	seed	Meltano command mapped to dbt seed.

Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:18, README.md:129-130

Source Freshness

Source freshness checks allow teams to monitor the latency of their data pipelines by querying the most recent timestamps in source tables against defined thresholds.

Freshness Invocation

The extension provides the freshness command, which serves as an alias for the dbt source freshness operation. This command is critical for operational monitoring within a Meltano-managed DAG.

# Command mapping in README.md
commands:
  freshness:
    args: source freshness
    description: Check the freshness of your source data.

Sources: README.md:133-134, README.md:208-209

Configuration and Environment Mapping

For both seeding and freshness checks to succeed, the extension must correctly map Meltano environment variables to the dbt profiles.yml. The extension uses the DBT_EXT_TYPE environment variable to determine which database-specific profile (e.g., Postgres, Snowflake, BigQuery) to initialize and use during invocation.

Relevant Settings

Setting Name	Environment Variable	Default	Purpose
type	DBT_EXT_TYPE	N/A	Specifies the database profile type.
project_dir	DBT_PROJECT_DIR	transform	The directory where dbt_project.yml resides.
skip_pre_invoke	DBT_EXT_SKIP_PRE_INVOKE	false	Skips clean/deps before running seed/freshness.

Sources: dbt_ext/extension.py:44-56, README.md:20-30

Summary

The meltano-dbt-ext extension provides a robust wrapper for dbt's seeding and source freshness capabilities. By automating environment preparation via pre_invoke hooks and mapping Meltano configurations to dbt profiles, it ensures that static data loading and data latency monitoring are seamlessly integrated into the Meltano ecosystem. These features are accessible via the dbt_invoker executable, allowing developers to maintain data quality and project dependencies with minimal manual intervention.

Sources: dbt_ext/extension.py:100-116, README.md:11-15