Troubleshooting and Support
Relevant source files
The following files were used as context for generating this wiki page:
The meltano-airflow-ext utility acts as a wrapper around Apache Airflow, facilitating its use within a Meltano project. Troubleshooting involves identifying issues in the initialization phase, environment configuration, and the dynamic generation of Airflow DAGs from Meltano schedules.
This extension manages the transition between Meltano's ELT/Job definitions and Airflow's orchestration engine. Support focuses on ensuring the AIRFLOW_HOME environment is correctly set, the database is migrated according to the specific Airflow version (2.x vs 3.x), and logs are correctly routed to the Meltano project structure.
Environment and Configuration Issues
Successful execution depends heavily on environment variables defining paths for logs, DAGs, and configuration files. If AIRFLOW_HOME is missing, the extension will fail immediately.
Critical Environment Variables
| Variable | Description | Source |
|---|---|---|
AIRFLOW_HOME | The directory for Airflow configuration and metadata. | airflow_ext/wrapper.py:35-41 |
AIRFLOW_CONFIG | Path to the airflow.cfg file. Defaults to AIRFLOW_HOME/airflow.cfg. | airflow_ext/wrapper.py:43 |
AIRFLOW__CORE__DAGS_FOLDER | Directory where Airflow looks for DAG files. | airflow_ext/wrapper.py:44-51 |
MELTANO_PROJECT_ROOT | Used to resolve absolute paths for logs and metadata. | README.md:27-77 |
Sources: airflow_ext/wrapper.py:35-51, README.md:27-77
Configuration Generation Flow
When meltano invoke airflow:initialize is called, the extension attempts to generate a default configuration. It strips leading warnings/logs from the airflow config list --defaults output to prevent ini file corruption.
The logic ensures that if Airflow emits deprecation notices to stdout, they do not end up inside the airflow.cfg file.
Sources: airflow_ext/wrapper.py:108-129
Database Initialization Errors
The extension handles database initialization differently depending on the Airflow version detected. Using the wrong command for the version installed will lead to failures during the initialize command.
- Airflow 2.x: Uses
airflow db init. - Airflow 3.x: Uses
airflow db migrate.
If the version cannot be determined or is unhandled, an error is logged.
Sources: airflow_ext/wrapper.py:132-160
DAG Generation Troubleshooting
DAGs are dynamically generated by meltano_dag_generator.py (copied from airflow_ext/files/orchestrate/meltano.py). If DAGs are not appearing in the Airflow UI, check the following:
Common Generator Failures
- Missing Meltano Binary: The generator looks for the
meltanoexecutable at.meltano/run/binrelative to the project root. If not found, it falls back to searching the systemPATH. - Missing Environment Variables: The generator requires
MELTANO_PROJECT_ROOTto properly set thecwdforBashOperatortasks. - Invalid Intervals: Schedules with
cron_intervalset toNoneor@onceare explicitly skipped and will not generate DAGs.
Log Location Reference
The default configuration routes different Airflow logs to specific subdirectories within the Meltano project to assist in debugging:
- General Logs:
$MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/logs - DAG Processor:
.../logs/dag_processor_manager/dag_processor_manager.log - Scheduler:
.../logs/scheduler
Sources: README.md:55-77
Support and Reporting
The extension uses structlog for structured logging. When encountering an "uncaught exception" during initialize or describe commands, the extension will log the full exception trace and suggest reporting it to the maintainers.
Diagnostic Command
To verify the extension's view of its own commands and capabilities:
meltano invoke airflow:describe --format json
Sources: airflow_ext/main.py:59-71, airflow_ext/wrapper.py:98-106
Testing Environment
If issues persist, you can replicate the environment using the project's tox configuration, which tests against both Airflow 2.9+ and 3.0+ across multiple Python versions (3.10 to 3.14).
Sources: tox.toml:1-15
Conclusion
Most issues in meltano-airflow-ext stem from incorrect environment pathing or version mismatches between the extension and the installed apache-airflow package. Ensuring AIRFLOW_HOME is correctly mapped to the Meltano project structure and running the initialize command are the primary steps for resolving setup failures.