Global Macros & Jinja Dispatch
Relevant source files
The following files were used as context for generating this wiki page:
- dbt-adapters/src/dbt/include/global_project/macros/adapters/relation.sql
- dbt-adapters/src/dbt/include/global_project/macros/adapters/schema.sql
- dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/column_helpers.sql
- dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql
- dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/on_schema_change.sql
- dbt-adapters/src/dbt/include/global_project/macros/python_model/python.sql
- AGENTS.md
The Global Macros and Jinja Dispatch system provides a standardized framework for cross-database compatibility within dbt. By utilizing a "dispatch" mechanism, dbt allows core logic—such as materializations, schema management, and incremental updates—to be defined once in a global project while allowing specific adapters (e.g., Postgres, Snowflake, BigQuery) to provide optimized implementations for their respective SQL dialects.
This system is central to the dbt-adapters monorepo architecture, enabling shared behavior across all first-party adapters while maintaining the flexibility required for specialized database features like BigQuery's partition_by or Snowflake's dynamic_table.
Sources: AGENTS.md:7-16, AGENTS.md:95-103
The Dispatch Mechanism
The dispatch system is the primary method for macro overriding. When a global macro is called, dbt uses adapter.dispatch to search for a specific implementation in the current adapter's namespace before falling back to a default__ implementation.
The diagram shows the logical flow of macro resolution during dbt execution.
Sources: AGENTS.md:105-112, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql:3-5
Implementation Patterns
Adapters override global behavior by prefixing the macro name with the adapter name (e.g., postgres__ or bigquery__). This allows the global project to define complex logic, like schema change processing, while delegating low-level operations like column type alteration to the adapter.
| Component | Global Macro Name | Purpose |
|---|---|---|
| Schema Changes | check_for_schema_changes | Identifies diffs between source and target relations. |
| Merge Logic | get_merge_sql | Generates DML for incremental model updates. |
| Column Management | diff_column_data_types | Compares data types between source and target columns. |
| Python Models | resolve_model_name | Sanitizes and renders model identifiers for Python scripts. |
Sources: dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/on_schema_change.sql:27-29, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql:15-18, dbt-adapters/src/dbt/include/global_project/macros/python_model/python.sql:1-3
Incremental Materialization Macros
Incremental models rely heavily on dispatched macros to manage table state. The system handles three primary areas: schema synchronization, column differencing, and DML generation.
Schema Synchronization Logic
The process_schema_changes macro orchestrates how dbt reacts when a model's SQL changes relative to the existing table in the database.
This sequence illustrates the interaction between global incremental macros and adapter-specific synchronization.
Sources: dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/on_schema_change.sql:110-155
Column Differentiation
To identify necessary changes, dbt-adapters provides utility macros for list manipulation and type checking:
diff_columns: Returns columns present in a source relation but missing from a target.diff_column_data_types: Dispatches to adapter-specific logic to determine if a column requires a type expansion (e.g., fromINTtoBIGINT).get_merge_update_columns: Filters columns to be updated during amergeoperation based on usermerge_update_columnsormerge_exclude_columnsconfigurations.
Sources: dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/column_helpers.sql:24-39, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/column_helpers.sql:43-45, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/column_helpers.sql:65-92
Python Model Support
Global macros also facilitate the generation of Python scripts for adapters supporting Python models (e.g., BigQuery, Snowflake, Spark). These macros construct the bridge between dbt's Jinja context and the Python execution environment.
Key Python Helper Macros
build_ref_function: Generates a Pythonref()function that maps dbt relation objects to rendered strings or JSON metadata.build_config_dict: Extracts dbt model configurations (likematerializedormeta) into a Python dictionary.py_script_postfix: Appends the necessary dbt logic classes (config,this,dbtObj) to the user-provided Python code.
Sources: dbt-adapters/src/dbt/include/global_project/macros/python_model/python.sql:5-32, dbt-adapters/src/dbt/include/global_project/macros/python_model/python.sql:48-69, dbt-adapters/src/dbt/include/global_project/macros/python_model/python.sql:84-118
Summary of Dispatchable SQL Generation
The following table summarizes core DML generation macros used in incremental strategies:
| Macro | Logic | Default Behavior |
|---|---|---|
get_merge_sql | Merge (Upsert) | Generates a standard SQL MERGE INTO statement with MATCHED and NOT MATCHED clauses. |
get_delete_insert_merge_sql | Delete+Insert | Executes a DELETE based on unique keys followed by a standard INSERT. |
get_insert_overwrite_merge_sql | Insert Overwrite | Uses MERGE with NOT MATCHED BY SOURCE to truncate partitions and insert new data. |
Sources: dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql:15-61, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql:64-96, dbt-adapters/src/dbt/include/global_project/macros/materializations/models/incremental/merge.sql:99-126
The Global Macros and Jinja Dispatch system ensures that the dbt-adapters framework remains extensible. By abstracting SQL patterns into the global project and providing a clear path for adapter-specific overrides via dispatch, dbt maintains a consistent user experience across wildly different database architectures.