Athena Adapter Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Athena Adapter Integration allows Meltano users to manage dbt projects targeting Amazon Athena. This integration is part of the meltano-dbt-ext utility, which wraps the dbt CLI to provide seamless configuration management via environment variables and Meltano's lifecycle hooks. The core of this integration lies in the specialized profiles.yml template and the extension's ability to initialize these profiles based on the DBT_EXT_TYPE setting.
Sources: README.md:1-10, dbt_ext/extension.py:34-45, dbt_ext/dbt_files/profiles/athena/profiles.yml:1-10
Architecture and Initialization
The integration follows a standardized lifecycle managed by the dbt class in extension.py. When a user runs the initialize command, the extension identifies the target adapter via the DBT_EXT_TYPE environment variable. For Athena, it locates the corresponding profile directory and deploys the profiles.yml to the project's profiles directory.
Initialization Workflow
The extension ensures that the transform directory and the specific profiles subdirectory are created before copying the adapter-specific configuration files.
Sources: dbt_ext/extension.py:118-144, dbt_ext/main.py:32-45
Configuration and Environment Variables
The Athena integration relies heavily on environment variables to populate the profiles.yml file. This allows for dynamic configuration across different Meltano environments (dev, staging, prod) without hardcoding credentials or connection details.
Profile Structure
The profiles.yml for Athena defines three standard outputs (dev, staging, prod), all referencing the same set of environment variables to ensure consistency across the Meltano lifecycle.
| Configuration Key | Environment Variable | Description |
|---|---|---|
type | N/A (Fixed: athena) | Specifies the dbt adapter type. |
s3_staging_dir | DBT_ATHENA_S3_STAGING_DIR | S3 path for staging query results. |
s3_data_dir | DBT_ATHENA_S3_DATA_DIR | S3 path where the data files are stored. |
region_name | DBT_ATHENA_REGION_NAME | AWS Region (e.g., us-east-1). |
schema | DBT_ATHENA_SCHEMA | The Athena schema (database) to use. |
database | DBT_ATHENA_DATABASE | The Glue Data Catalog database name. |
work_group | DBT_ATHENA_WORK_GROUP | Athena Workgroup for query execution. |
Sources: dbt_ext/dbt_files/profiles/athena/profiles.yml:12-38
Execution Logic
The extension wraps the dbt binary using an Invoker. Before executing standard dbt commands (like run or test), the extension optionally performs "pre-invoke" steps.
Pre-invoke Operations
Unless DBT_EXT_SKIP_PRE_INVOKE is set to true, the extension automatically executes dbt clean and dbt deps before the primary command. This ensures that the environment is clean and all dbt packages are installed.
Project Integration
The Athena integration utilizes a standard dbt_project.yml file bundled within the extension. This file defines the project name and the directory structure for models, seeds, and macros.
name: my_meltano_project
version: '1.0'
profile: meltano
config-version: 2
clean-targets:
- target
- dbt_packages
- logs
The profile: meltano setting in dbt_project.yml matches the top-level key in the Athena profiles.yml, linking the project to the environment-aware connection settings.
Sources: dbt_ext/dbt_files/bundle/transform/dbt_project.yml:1-25, dbt_ext/dbt_files/profiles/athena/profiles.yml:7
Summary
The Athena Adapter Integration provides a robust way to run dbt against AWS Athena within a Meltano project. By utilizing the meltano-dbt-ext framework, it automates the deployment of Athena-specific profiles and leverages environment variables for secure, environment-specific configurations. The integration ensures that standard dbt workflows, including dependency management and workspace cleanup, are maintained through the extension's pre-invoke hooks.