dbt-spark Adapter
Relevant source files
The following files were used as context for generating this wiki page:
The dbt-spark adapter is a plugin for dbt that enables transformation workflows on Apache Spark and Databricks. It extends the SQLAdapter base class provided by the dbt-adapters framework, supporting both open-source Spark distributions and managed environments like Databricks.
Sources: CLAUDE.md:215, dbt-spark/pyproject.toml:7
Architecture and Connection Methods
The adapter supports multiple connection protocols to interface with Spark clusters or SQL endpoints. These methods are configured within the dbt profile and dictate how dbt communicates with the compute engine.
Supported Connection Methods
| Method | Description |
|---|---|
thrift | Connects via the Spark Thrift Server (standard for open-source Spark). |
odbc | Uses the Databricks ODBC driver for connectivity. |
http | Interfaces with Databricks HTTP endpoints. |
databricks | Utilizes the specialized databricks-sql-connector. |
Sources: CLAUDE.md:219-222, dbt-spark/CONTRIBUTING.md:16-24
Component Relationship
The following diagram illustrates how the Spark adapter interacts with different Spark implementations:
The diagram shows the abstraction layer provided by the adapter, allowing the same dbt models to run across various Spark-compatible backends by swapping connection methods.
Sources: CLAUDE.md:215-222, dbt-spark/CONTRIBUTING.md:16-24
Materialization Logic
The adapter provides specialized implementations for standard dbt materializations, optimized for Spark features like Delta Lake, Iceberg, and partitioning.
Table Materialization
Table materialization in dbt-spark handles the creation of physical tables. It includes logic to perform "Create or Replace" operations specifically for Delta and Iceberg formats to ensure high availability and prevent table downtime during replacement.
Sources: dbt-spark/src/dbt/include/spark/macros/materializations/table.sql:1-35
Incremental Materialization
The incremental materialization supports several strategies optimized for big data workloads. It handles complex logic for schema changes and partition management.
The flow depicts the decision path for incremental updates, including specialized handling for insert_overwrite and merge.
Sources: dbt-spark/src/dbt/include/spark/macros/materializations/incremental/incremental.sql:1-75
Incremental Strategies and Configurations
| Strategy | Description | Supported Formats |
|---|---|---|
append | Default strategy; appends new rows to the existing table. | All |
insert_overwrite | Overwrites specific partitions; requires partition_by. | Parquet, Delta, etc. |
merge | Performs atomic upserts based on a unique key. | Delta Lake |
Sources: dbt-spark/src/dbt/include/spark/macros/materializations/incremental/incremental.sql:2-10, CLAUDE.md:231-235
Python Model Support
dbt-spark supports Python-based models, allowing users to define transformations using PySpark, Pandas, or Koalas. The adapter automatically generates the boilerplate code required to wrap the user's Python function into a Spark job that saves the resulting DataFrame.
Execution Flow for Python Models
This diagram outlines the process where dbt submits a dynamically constructed script to Spark, which then handles the conversion and persistence of the data.
Sources: dbt-spark/src/dbt/include/spark/macros/materializations/table.sql:56-104
Configuration Options
The adapter supports a variety of Spark-specific configurations that can be applied at the model or project level.
| Configuration | Type | Purpose |
|---|---|---|
file_format | string | Storage format: delta, parquet, iceberg, hudi, etc. |
partition_by | list | Columns used for physical data partitioning. |
buckets | integer | Number of buckets for bucketing. |
bucket_columns | list | Columns used for bucketing. |
location | string | External S3/ADLS/GCS path for the table. |
tblproperties | dict | Key-value pairs for Spark table properties. |
Sources: CLAUDE.md:224-237, dbt-spark/src/dbt/include/spark/macros/materializations/incremental/incremental.sql:10-25
Technical Dependencies
The adapter relies on several Python libraries to facilitate its various connection methods:
dbt-adapters: Base framework for dbt adapters.PyHive: Required forthriftconnections.thrift: Protocol for Spark Thrift Server communication.databricks-sql-connector: Required for modern Databricks SQL connectivity.pyodbc: Optional; required if using theodbcmethod.pyspark: Required for local Spark session execution.
Sources: dbt-spark/pyproject.toml:35-50, CLAUDE.md:243-245
Conclusion
The dbt-spark adapter provides a bridge between dbt's SQL-centric modeling and the distributed computing power of Apache Spark. By abstracting connectivity protocols and optimizing materializations for big data formats like Delta Lake and Iceberg, it allows data teams to manage complex Spark transformations using standard dbt practices.