CI/CD & airbyte-ci

Relevant source files

The following files were used as context for generating this wiki page:

• docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/writing-docs.md
• docusaurus/platform_versioned_docs/version-1.8/contributing-to-airbyte/writing-docs.md
• docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
• docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/resources/qa-checks.md
• docusaurus/platform_versioned_docs/version-1.8/contributing-to-airbyte/resources/qa-checks.md

The CI/CD (Continuous Integration and Continuous Deployment) infrastructure in Airbyte is designed to ensure that code contributions, particularly for connectors and documentation, meet high standards of quality, security, and reliability. Central to this process is airbyte-ci, a custom tooling suite used to run static analysis, tests, and deployment workflows. These workflows are integrated with GitHub Actions and other CI providers like Vercel for documentation.

The system enforces rigorous QA checks on connectors to ensure they follow best practices before being released to registries like DockerHub or PyPI. For documentation, the CI pipeline automates linting and preview builds, allowing contributors to validate changes before merging into the master branch.

Automated QA Checks and Static Analysis

Airbyte employs an extensive set of automated QA checks that run within the CI/CD pipeline. These checks are categorized into documentation, metadata, packaging, assets, security, and testing. They are mandatory for connectors seeking release to registries.

Documentation and Metadata Enforcement

CI workflows verify that every new connector version or breaking change is accompanied by the necessary user-facing documentation and migration guides. Metadata files (metadata.yaml) are strictly validated against a schema to ensure the connector registry remains accurate.

Security and Packaging Standards

The CI system enforces specific packaging and security rules:

• Dependency Management: Python connectors must use Poetry for reproducible installs.
• Licensing: Connectors must be licensed under MIT or Elv2.
• Base Images: Python connectors are prohibited from using custom Dockerfiles; they must declare a maintained base image in metadata.yaml.
• Network Security: Connectors must exclusively use HTTPS for external requests.

Sources: version-1.8/contributing-to-airbyte/resources/qa-checks.md:195-201, version-1.8/contributing-to-airbyte/resources/qa-checks.md:240-246, version-1.8/contributing-to-airbyte/resources/qa-checks.md:261-267

Documentation CI/CD Workflows

Documentation updates follow a specialized CI/CD path involving linting tools and automated deployment platforms.

Reviewdog and Linting

The Reviewdog workflow runs automatically on pull requests that modify Markdown files in the docs/ directory. It integrates two primary linters:

1. Vale: Lints for style, grammar, and passive language based on the Google developer documentation style guide.
2. MarkDownLint: Lints for Markdown structural issues, such as skipped heading levels.

Violations are posted as annotations directly on the GitHub "Files Changed" page.

Deployment and Previews

Airbyte uses Vercel for documentation hosting and deployment.

• Previews: Vercel automatically builds a preview of the documentation site for every pull request. For community contributors, an Airbyte employee must authorize the build.
• Production Deployment: Once changes are merged into the master branch, Vercel automatically deploys the updated documentation to docs.airbyte.com. This process typically takes 5-10 minutes.

Sources: version-2.0/contributing-to-airbyte/writing-docs.md:435-442, version-2.0/contributing-to-airbyte/writing-docs.md:456-460

Key CI/CD Components Summary

Component	Description	Relevance
metadata.yaml	Configuration file at the connector root.	Used to build the connector registry and validate CI standards.
Reviewdog	GitHub Actions workflow.	Automates the reporting of linter violations in PRs.
Poetry	Dependency management tool.	Required for all Python connectors to ensure reproducible installs in CI.
Vercel	Deployment platform.	Manages preview and production builds for documentation.
acceptanceTests	Suite in metadata.yaml.	Required for medium-to-high use connectors to ensure functional correctness.

Sources: version-1.8/contributing-to-airbyte/resources/qa-checks.md:144-148, version-1.8/contributing-to-airbyte/resources/qa-checks.md:195-199, version-2.0/contributing-to-airbyte/writing-docs.md:220-225, version-1.8/contributing-to-airbyte/resources/qa-checks.md:280-285

Versioning and Release Management

The CI/CD system supports multi-version documentation for the platform. This allows maintaining "frozen" sets of documentation for released versions while the "Next" version continues to receive updates.

Release Workflow:

1. Branching: A new branch is created off master.
2. Versioning Command: Running pnpm run docusaurus docs:version:platform <version> creates a physical copy of the docs and sidebars in the /docusaurus/ folder.
3. CI Validation: The PR must pass all CI checks, including broken link detection and linting, before merging.

Sources: version-1.8/contributing-to-airbyte/writing-docs.md:472-485, version-2.0/contributing-to-airbyte/writing-docs.md:502-515

The CI/CD and airbyte-ci systems provide a robust framework that automates the enforcement of Airbyte's technical standards. By offloading static analysis and deployment to automated pipelines, the project maintains high velocity while ensuring that every released connector or documentation page is secure, reliable, and properly formatted.