CI/CD & GitHub Actions
Relevant source files
The following files were used as context for generating this wiki page:
The CI/CD infrastructure for dbt Core v2.0 (Fusion) is designed to manage automated testing, pull request triage, and complex multi-platform release pipelines. It relies on a combination of GitHub Actions for workflow orchestration and a dedicated internal tool, dbt-ci, for executing release-specific operations such as version bumping, artifact packing, and distribution to PyPI and Homebrew.
Sources: CONTRIBUTING.md, crates/dbt-ci/README.md:1-10
Pull Request Lifecycle and Automation
Every contribution to dbt Core follows a structured CI path managed through labels and automated checks. Pull requests are automatically triaged and labeled to reflect their origin and status.
PR Triage and Status Labels
When a PR is opened, it is automatically assigned labels that drive the internal build and review process.
| Label | Meaning |
|---|---|
community / source:community | Identifies changes from external contributors. |
ci:approve-public-fork-ci | Required for fork PRs to access repository secrets; must be re-applied after every commit. |
review-status: in-review | Change has synced successfully and is currently under active review. |
review-status: sync-failed | The sync between internal and public repositories failed (e.g., merge conflict). |
review-status: merged-upstream | Change has been merged into the internal source of truth. |
Skip Changelog | Bypasses the requirement for a changie entry. |
Sources: CONTRIBUTING.md:46-75
Automated Checks
The CI pipeline performs several automated validations:
- Changelog Check: Ensures that a new entry has been created using
changie new. - dbt Labs CI: Surfaces general build results and status checks directly on the PR.
- Sync Status: Reflects whether the change has been successfully applied to the upstream development branch.
Sources: CONTRIBUTING.md:55-75, CONTRIBUTING.md:83-110
PR Workflow Diagram
The following diagram illustrates the lifecycle of a pull request from submission to merge.
This diagram shows the logic gate for fork PRs and the automated changelog enforcement.
Sources: CONTRIBUTING.md:46-75
The dbt-ci Release Tool
dbt-ci is a Rust-based utility located in crates/dbt-ci that serves as the engine for release-pipeline commands. It is typically invoked via the alias cargo ci.
Subcommands and Functionality
| Command | Action |
|---|---|
bump-cargo-version | Updates [workspace.package].version and refreshes Cargo.lock. |
pypi pack | Generates py3-none-{platform} wheels for pre-built binaries. |
pypi publish | Uploads artifacts to PyPI (supports sdist and wheel modes). |
homebrew render | Generates a Ruby Formula for Homebrew from release tarballs. |
homebrew publish | Commits and pushes the formula to a Homebrew tap repository. |
Sources: crates/dbt-ci/README.md:5-24, crates/dbt-ci/Cargo.toml
Artifact Distribution Pipelines
PyPI Distribution Logic
The dbt-ci tool handles publishing to multiple environments: Staging (AWS CodeArtifact), Prod (PyPI), and Test-PyPI.
Distribution Modes
- Wheel Upload Mode: Discovers local wheels in
target/wheelsand uploads them directly. - Sdist Mode: Builds a "tiny" source distribution (
.tar.gz) containing a PEP 517 backend and anassets.jsonmanifest. This manifest maps platform tags to specific wheel filenames and SHA256 hashes, allowing the sdist to function as a "download-at-install" proxy.
Sources: crates/dbt-ci/src/publish.rs:135-175, crates/dbt-ci/src/sdist.rs:1-15
Release Sequence Diagram
The release process involves building binaries for various targets and then orchestrating their publication.
This diagram depicts the typical flow of a release, where binaries are built before being "packed" into Python wheels and published.
Sources: crates/dbt-ci/README.md:41-65, crates/dbt-ci/src/publish.rs:195-230
Homebrew Formula Management
The Homebrew pipeline renders formulas based on release tarballs. It supports multiple architectures including macOS (ARM/Intel) and Linux (ARM/Intel).
- Rendering: Uses
pyproject.tomlmetadata (homepage, license, description) and hashes from release tarballs to build the.rbformula. - Idempotency: The
publishcommand is idempotent; it will not push a new commit if the generated formula is byte-identical to the existing one in the tap.
Sources: crates/dbt-ci/src/homebrew/render.rs:13-40, crates/dbt-ci/src/homebrew/render.rs:166-210
Configuration and Environment
The CI system relies on specific environment variables to authenticate with distribution platforms.
| Environment | Variables |
|---|---|
| AWS CodeArtifact | DBT_PYPI_STAGING_DOMAIN, _DOMAIN_OWNER, _REGION, _REPOSITORY, _PROFILE |
| PyPI (Prod) | DBT_PYPI_PROD_TOKEN |
| PyPI (Test) | DBT_PYPI_TEST_TOKEN |
| Homebrew | HOMEBREW_TAP_REPO_TOKEN |
Sources: crates/dbt-ci/src/publish.rs:36-70, crates/dbt-ci/README.md:33-39
Release Sdist Structure
The tiny sdist is a critical architectural component. It ensures that users can pip install dbt-core while receiving the pre-compiled Rust binary appropriate for their system.
{
"name": "dbt-core",
"version": "2.0.0",
"base_url": "https://github.com/dbt-labs/dbt-core/releases/download/v2.0.0%22%2C
"wheels": {
"manylinux_2_28_x86_64": {
"filename": "dbt_core-2.0.0-py3-none-manylinux_2_28_x86_64.whl",
"sha256": "..."
}
}
}
Sources: crates/dbt-ci/src/sdist.rs:24-40, crates/dbt-ci/src/sdist.rs:130-155
Summary
The dbt Core CI/CD system automates the transition from code change to cross-platform distribution. By leveraging the dbt-ci tool, the project maintains a strict release sequence that handles versioning, multi-architecture binary packing, and tiered publication to both AWS and public registries. The use of a manifest-driven sdist allows for optimized installation of Rust-based tools within the Python ecosystem.