dbt-athena Adapter

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• AGENTS.md
• CLAUDE.md
• dbt-adapters/src/dbt/adapters/events/types.py
• dbt-tests-adapter/src/dbt/tests/adapter/basic/test_docs_generate.py

The dbt-athena adapter is a specialized plugin within the dbt ecosystem that enables dbt to interact with Amazon Athena. It translates dbt's transformation logic into Athena-compliant SQL and manages the underlying AWS infrastructure requirements, such as S3 storage and Lake Formation permissions.

As part of the dbt-adapters monorepo, it extends the SQLAdapter base class to provide SQL-based transformation capabilities while integrating with AWS-specific services like boto3 and pyathena. It is designed to handle the unique requirements of a serverless query service, including the management of S3 staging directories and data partitioning.

Sources: README.md:16-30, AGENTS.md:14-23, CLAUDE.md:214-230

Architecture and Core Components

The adapter architecture follows a modular structure common to dbt v1 adapters. It utilizes a combination of Python logic for session management and Jinja-templated SQL macros for data manipulation.

Component Relationship

The following diagram illustrates how the dbt-athena adapter sits between dbt-core and the AWS ecosystem.

Configuration and Credentials

Connecting dbt to Athena requires specific AWS configuration parameters. These are defined in the Credentials class and typically supplied via a profiles.yml file.

Required Connection Parameters

Parameter	Type	Description
s3_staging_dir	String	S3 path where Athena stores query results.
s3_tmp_table_dir	String	S3 path for temporary table data.
region_name	String	AWS Region (e.g., us-east-1).
database	String	The AWS Glue database name.
work_group	String	Athena workgroup for query execution and cost control.
aws_profile_name	String	Named AWS profile from local credentials file.
poll_interval	Integer	Seconds to wait between checking query status.

Sources: CLAUDE.md:231-235

Feature Support and Implementation

The dbt-athena adapter implements several features unique to the Athena environment, particularly regarding table formats and partitioning.

Table Formats and Materializations

The adapter supports various table formats, including Hive and Apache Iceberg. This is controlled through dbt configuration blocks.

{{ config(
    materialized='table',
    table_type='iceberg',
    format='parquet',
    partitioned_by=['year', 'month'],
    external_location='s3://my-bucket/path/'
) }}

Example of an Athena-specific model configuration.

Key Logic Modules

• impl.py: Contains the core AthenaAdapter implementation.
• s3.py: Manages S3 object lifecycle, including cleaning up orphaned data files when tables are dropped or replaced.
• lakeformation.py: Handles integration with AWS Lake Formation for fine-grained access control.
• python_submissions.py: Manages the execution of Python models if supported by the Athena environment.

Sources: CLAUDE.md:214-230, AGENTS.md:104-109

Testing Framework

The adapter is tested using the dbt-tests-adapter suite to ensure it meets the standard "basic" requirements for dbt adapters, such as materializations and documentation generation.

Integration Test Flow

The testing process involves setting up a live connection to AWS to verify SQL execution and S3 side effects.

Event Logging

The adapter utilizes the dbt.adapters.events system to provide debug and informational logs during execution. Specific events are triggered for connection acquisition, query status, and cache actions.

Event Class	Code	Level	Description
NewConnection	E005	Debug	Logged when a new Athena session is initialized.
SQLQueryStatus	E017	Debug	Reports the status and elapsed time of an Athena query.
CatalogGenerationError	E040	Warn	Triggered if metadata retrieval from Glue fails.

Sources: dbt-adapters/src/dbt/adapters/events/types.py:73-77, 186-191, 355-360

Conclusion

The dbt-athena adapter provides a bridge between dbt's transformation workflows and AWS's serverless data lake architecture. By managing S3 paths, Glue metadata, and Athena-specific SQL syntax, it allows users to treat their data lake as a standard data warehouse while leveraging the scalability of AWS.

Sources: CLAUDE.md:214-235, AGENTS.md:14-23