Relation Representation & Metadata

Relevant source files

The following files were used as context for generating this wiki page:

• dbt-adapters/src/dbt/adapters/base/relation.py
• dbt-adapters/src/dbt/adapters/contracts/relation.py
• dbt-adapters/src/dbt/adapters/relation_configs/config_base.py
• dbt-spark/src/dbt/adapters/spark/impl.py
• dbt-spark/src/dbt/include/spark/macros/adapters.sql
• dbt-adapters/src/dbt/include/global_project/macros/adapters/metadata.sql

Introduction

In the dbt-adapters ecosystem, a Relation is the fundamental abstraction representing a database object, such as a table, view, or materialized view. This representation encompasses the logical path (database, schema, identifier), the object type, and associated metadata used for DDL generation and cataloging. The system ensures that these representations are consistent across different database adapters while allowing for platform-specific extensions, such as Spark-specific provider information or BigQuery partitioning configs.

Metadata management involves retrieving existing object structures from the data warehouse to populate dbt's internal cache and generate documentation. This process includes listing relations, describing columns, and capturing extended properties like table statistics or partitioning details. The integration between Python-based adapter logic and Jinja-based SQL macros allows dbt to dynamically interact with diverse information schemas.

Core Components of Relation Representation

The Relation Class Hierarchy

Relations are primarily defined through the BaseRelation class and its contract-based counterpart RelationConfig. The BaseRelation handles the rendering of identifiers for SQL queries, while RelationConfig ensures data integrity for serializing relation states.

Sources: dbt-adapters/src/dbt/adapters/contracts/relation.py, dbt-adapters/src/dbt/adapters/base/relation.py, dbt-spark/src/dbt/adapters/spark/impl.py:88-100

Relation Configuration and Changes

The framework uses RelationConfigBase to manage the inspection of existing relations and RelationConfigChange to track delta operations between the desired state (from dbt code) and the current state (from the warehouse).

Component	Description
RelationConfigBase	A base class providing from_dict() for initializing configurations from dictionary-like objects.
RelationConfigChange	Restricted class containing an action (e.g., 'drop', 'create') and a context defining the new state.
RelationConfigValidationMixin	Provides validation_check logic to ensure configurations meet specific criteria before application.

Sources: dbt-adapters/src/dbt/adapters/relation_configs/README.md, dbt-adapters/src/dbt/adapters/relation_configs/config_base.py

Metadata Retrieval Logic

Metadata retrieval is performed through a series of dispatched macros and adapter methods. The workflow typically begins with listing all relations in a schema and then drilling down into specific column metadata.

Relation Discovery Flow

Adapters implement list_relations_without_caching to query the platform's information schema or specialized commands (like SHOW TABLES in Spark).

Column Metadata Extraction

For each relation, dbt fetches column details including names, data types, and comments. In Spark, this involves parsing the output of DESCRIBE EXTENDED, which contains both column definitions and table-level metadata like "Owner" or "Statistics".

Spark Column Metadata Parsing:

1. Separator Identification: The parser looks for the # Detailed Table Information marker in the DESCRIBE output.
2. Column Extraction: Rows before the separator are treated as standard columns.
3. Metadata Extraction: Rows after the separator are mapped to table-level attributes (e.g., Owner, Statistics).

Sources: dbt-spark/src/dbt/adapters/spark/impl.py:284-332, dbt-spark/src/dbt/include/spark/macros/adapters.sql:280-284

Metadata Macros and SQL Interfaces

The interaction with the data warehouse is abstracted through Jinja macros defined in global_project and overridden in specific adapter projects.

Key Metadata Macros

Macro	Purpose	Implementation Details
get_catalog	Retrieves the full catalog for a set of schemas.	Uses information_schema.SCHEMATA by default.
list_schemas	Lists all schemas (databases) in the current environment.	Dispatched to adapter-specific SQL.
check_schema_exists	Validates if a specific schema exists.	Queries information_schema.SCHEMATA.
list_relations_without_caching	Lists tables/views in a specific schema.	Base implementation raises NotImplementedError.

Sources: dbt-adapters/src/dbt/include/global_project/macros/adapters/metadata.sql:1-96

Spark Relation Creation Metadata

Spark relations require specific DDL clauses captured in the metadata model, such as file_format, location, and tblproperties.

{% macro spark__create_table_as(temporary, relation, compiled_code, language='sql') -%}
  ...
  {{ file_format_clause() }}
  {{ options_clause() }}
  {{ tblproperties_clause() }}
  {{ partition_cols(label="partitioned by") }}
  {{ location_clause() }}
  as {{ compiled_code }}
{%- endmacro -%}

Sources: dbt-spark/src/dbt/include/spark/macros/adapters.sql:149-178

Summary of Relation Metadata Structures

The following table summarizes the data model for relation metadata as represented in the Spark adapter, which is indicative of the complexity managed by the system.

Field	Type	Description
table_database	String	The database name (often None in Spark).
table_schema	String	The namespace/schema of the relation.
table_name	String	The identifier of the relation.
column_name	String	The name of the specific column.
column_type	String	The data type (e.g., string, bigint).
table_owner	String	Extracted from Spark's Detailed Table Information.
stats:rows:value	Integer	Estimated row count from statistics metadata.

Sources: dbt-spark/src/dbt/adapters/spark/impl.py:108-127

Conclusion

Relation Representation & Metadata forms the backbone of dbt's ability to remain "aware" of the data warehouse state. By combining strict Python contracts for object identifiers with flexible Jinja-based metadata queries, dbt-adapters provides a consistent interface for complex transformations across diverse platforms like PostgreSQL, BigQuery, and Spark. The use of RelationConfig and BaseRelation allows for a structured approach to DDL generation, while specific adapter implementations handle the nuances of platform-specific metadata formats.