External Catalog Integrations
Relevant source files
The following files were used as context for generating this wiki page:
External Catalog Integrations provide a framework within dbt adapters to support external table formats, primarily Apache Iceberg, by interfacing with various catalog providers. This system enables adapters to manage table metadata and DDL generation for tables stored in external locations while maintaining a consistent interface within dbt's model configuration.
The integration layer allows dbt to communicate with catalogs such as Snowflake's built-in Iceberg catalog, Iceberg REST servers, AWS Glue, and BigQuery's BigLake Metastore. By abstracting the catalog logic, the system supports complex workflows like Cross-Database operations and writes to External Volumes.
Sources: CLAUDE.md, AGENTS.md, dbt-snowflake/src/dbt/adapters/snowflake/impl.py:126-134
Architecture and Core Components
The architecture relies on a plugin-based system located in dbt-adapters/src/dbt/adapters/catalogs/. Each adapter registers its supported catalog integrations, which are then managed by a central client.
Primary Classes
| Class | Purpose |
|---|---|
CatalogIntegration | Abstract base class that defines the interface for specific catalog implementations (e.g., Iceberg REST). |
CatalogIntegrationConfig | A protocol defining the user-provided configuration for a catalog integration. |
CatalogRelation | A protocol for catalog-specific relation data, used to build DDL for external tables. |
CatalogIntegrationClient | A registry that manages multiple integrations and resolves them by name during model execution. |
Data Flow for Catalog Resolution
When a model is configured with a catalog_name, the adapter uses the CatalogIntegrationClient to retrieve the corresponding integration and build a CatalogRelation. This object contains the metadata (like external volumes or table formats) required to generate the final SQL.
Snowflake Implementation
The Snowflake adapter implements several specific integrations to handle its Iceberg and Horizon capabilities. It maps user-facing catalog types to internal integration logic.
Supported Integrations
BuiltInCatalogIntegration: Handles Snowflake-managed Iceberg tables.InfoSchemaCatalogIntegration: Standard Snowflake tables using Information Schema.IcebergRestCatalogIntegration: Supports external Iceberg catalogs via REST, including AWS Glue and Unity Catalog.
Sources: dbt-snowflake/src/dbt/adapters/snowflake/impl.py:126-130, dbt-snowflake/src/dbt/adapters/snowflake/impl.py:157-162
Configuration Mapping
The adapter provides a translation layer for "Catalogs V2" configurations, mapping external properties to Snowflake-specific internal fields.
_V2_TO_V1_TYPE: ClassVar[Dict[str, str]] = {
"horizon": "BUILT_IN",
"glue": "ICEBERG_REST",
"iceberg_rest": "ICEBERG_REST",
"unity": "ICEBERG_REST",
}
_LINKED_FIELD_MAP: ClassVar[Dict[str, str]] = {
"catalog_database": "catalog_linked_database",
}
Sources: dbt-snowflake/src/dbt/adapters/snowflake/impl.py:149-156
Integration with dbt Project Configuration
Users define external catalogs in dbt_project.yml or a dedicated catalogs.yml file. This configuration specifies the write integrations and their associated adapter properties.
Example Configuration (catalogs.yml)
catalogs:
- name: my_iceberg_catalog
active_write_integration: my_glue_integration
write_integrations:
- name: my_glue_integration
catalog_type: iceberg_rest
table_format: iceberg
adapter_properties:
catalog_linked_database: "my_external_db"
catalog_linked_database_type: "glue"
external_volume: "my_s3_volume"
Sources: CLAUDE.md, dbt-snowflake/tests/functional/adapter/catalog_integrations/test_iceberg_rest_catalog_integrations.py:68-87
Model Configuration Usage
To use an external catalog, the catalog_name is provided in the model's config block. This triggers the adapter to use the external catalog's DDL logic instead of standard CREATE TABLE AS (CTAS).
{{ config(
materialized='table',
catalog_name='basic_iceberg_rest_catalog'
) }}
select 1 as id
DDL Generation Logic
The system utilizes the build_catalog_relation method to inject catalog-specific metadata into the Jinja context. This metadata is used by macros to determine if a table should be created as an Iceberg table.
Configurable Properties for Iceberg
Specific implementations like IcebergRestCatalogIntegration support various adapter properties that influence the generated DDL:
max_data_extension_time_in_daystarget_file_sizeauto_refreshstorage_serialization_policy
Sources: dbt-snowflake/tests/functional/adapter/catalog_integrations/test_iceberg_rest_catalog_integrations.py:79-84, dbt-snowflake/tests/functional/adapter/catalog_integrations/test_catalogs_v2.py:53-61
Incremental Materialization Support
External catalogs support incremental strategies such as merge and insert_overwrite. On Snowflake, this often involves managing temporary relations that match the external table's format to ensure compatibility during the sync process.
Sources: dbt-snowflake/tests/functional/adapter/catalog_integrations/test_iceberg_rest_catalog_integrations.py:20-33, dbt-snowflake/CHANGELOG.md:88-89