Local Development Setup
Relevant source files
The following files were used as context for generating this wiki page:
This page provides technical guidance for setting up a local development environment for the dbt-adapters monorepo. The repository contains the base framework, a shared test suite, and multiple first-party adapters (PostgreSQL, Redshift, Snowflake, BigQuery, Spark, and Athena). Because this is a monorepo, each package is designed to be installed independently but often relies on local changes made to the base dbt-adapters package during development.
The development workflow centers on hatch, which manages virtual environments, dependencies, and task orchestration. Contributors typically work within a specific adapter directory while maintaining an editable link to the core adapter framework.
Sources: CONTRIBUTING.md:20-29, AGENTS.md:5-15, CLAUDE.md:3-12
Prerequisites and Core Utilities
Before setting up specific packages, several system-level utilities must be available. These tools handle package management, version control, and quality assurance.
| Utility | Purpose |
|---|---|
pip | Python package installer |
hatch | Environment manager and build tool |
git | Version control |
changie | Changelog entry management |
pre-commit | Git hooks for code quality |
Sources: CONTRIBUTING.md:10-18, AGENTS.md:22-30
Environment Initialization
Development is conducted within specific adapter directories rather than the repository root. The initialization process creates an editable environment where changes to one package (e.g., dbt-adapters) are immediately reflected in another (e.g., dbt-postgres).
Setup Workflow
The standard initialization sequence involves navigating to the target adapter and running the setup script via hatch.
The diagram shows the logical progression from navigating to a package to initializing the hatch environment.
Sources: AGENTS.md:22-30, CLAUDE.md:23-38
The hatch run setup Command
This command performs several automated tasks defined in the project's configuration:
- Installs the current package in editable mode.
- Installs development dependencies (e.g.,
pytest,mypy). - Installs other repository packages (
dbt-adapters,dbt-tests-adapter) in editable mode. - Registers
pre-commithooks. - Generates a
test.envfile fromtest.env.exampleif it does not exist.
Sources: CONTRIBUTING.md:33-41, dbt-adapters/hatch.toml:17-18
Cross-Package Development
Since the repository is a monorepo, implementing a feature often requires changes across multiple packages. For example, a new feature in dbt-postgres might require updates to the dbt-adapters base framework.
Editable Mode Relationships
By default, the environment is configured so that local changes in dbt-adapters are reflected in the virtual environment of any adapter (e.g., dbt-redshift). This ensures that PR checks function correctly without manually updating requirements to point to feature branches.
Sources: CONTRIBUTING.md:20-29, CONTRIBUTING.md:76-80
Developing Against External Core Branches
If changes are required in dbt-common or dbt-core, developers must manually install these external dependencies in editable mode within the hatch environment:
hatch run pip install -e ../../dbt-common
Note: This assume the external repo is cloned alongside dbt-adapters.
Sources: CONTRIBUTING.md:82-90
Testing Infrastructure
Tests are categorized into three levels: code quality, unit tests, and integration tests.
Code Quality and Unit Testing
Code quality checks (Black, Flake8, MyPy) are configured at the repository level but can be run within any package environment. Unit tests do not require a database and test Python logic directly.
# Run quality checks
hatch run code-quality
# Run all unit tests
hatch run unit-tests
# Run a specific unit test file
hatch run unit-tests tests/unit/test_logic.py
Sources: CONTRIBUTING.md:113-128, AGENTS.md:43-53
Integration Testing
Integration tests require a live database connection. Configuration is managed via environment variables stored in a test.env file.
| Adapter | Test Requirements | Configuration File |
|---|---|---|
| General | Live database instance | test.env |
dbt-spark | Dagger (containers) | test.env.example |
dbt-redshift | Redshift cluster | test.env.example |
For dbt-spark, local development involves a docker-compose environment providing a Spark Thrift server and Postgres Hive Metastore.
Sources: CONTRIBUTING.md:130-138, dbt-spark/CONTRIBUTING.md:9-15, dbt-spark/README.md:43-55
Dependency Management
Package dependencies and scripts are defined in hatch.toml and pyproject.toml files within each package directory.
Example: hatch.toml Environment Scripts
# dbt-adapters/hatch.toml
[envs.default.scripts]
setup = "pre-commit install"
code-quality = "pre-commit run --all-files"
unit-tests = "python -m pytest {args:tests/unit}"
Sources: dbt-adapters/hatch.toml:17-21
External Catalog Integration Development
For developers working on external table formats (e.g., Apache Iceberg), the system uses a plugin architecture located in dbt-adapters/src/dbt/adapters/catalogs/. Implementing a new catalog requires defining a CatalogIntegration and registering it in the adapter's __init__.py.
Sources: AGENTS.md:96-107, CLAUDE.md:183-195
Conclusion
Setting up the local development environment for dbt-adapters relies heavily on hatch to maintain the integrity of the monorepo's editable cross-references. By following the standardized setup workflow and utilizing the provided testing suites, developers can ensure that changes to the base framework and individual adapters are validated across the entire dependency chain.
Sources: CONTRIBUTING.md, AGENTS.md