Docker Deployment

Relevant source files

The following files were used as context for generating this wiki page:

• airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md
• docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
• airbyte-integrations/developer-docs/destinations/preflight-checklist.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
• docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md

Docker deployment serves as the primary mechanism for packaging and executing Airbyte components, specifically its vast ecosystem of connectors. Within the Airbyte architecture, each connector is encapsulated as a standalone Docker image, ensuring environment isolation, reproducibility, and a consistent interface for the platform to interact with various data sources and destinations.

The deployment process involves a standardized lifecycle: from local environment setup and configuration using Gradle plugins to image assembly and automated quality assurance checks. This modular approach allows the Airbyte platform to scale by dynamically spinning up connector containers as needed for synchronization tasks.

Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:1-20, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md:10-15

Connector Infrastructure and Packaging

Every Airbyte connector follows a strict packaging convention to be compatible with the platform. This includes a metadata.yaml file for registry management and specific Docker base images maintained by Airbyte to ensure security updates and performance standards.

The metadata.yaml Configuration

The metadata.yaml file is the source of truth for the connector's identity within the Docker ecosystem. It defines the image repository, version tags, and base image requirements.

Field	Description	Requirement
connectorType	Type of connector (source or destination)	Required
dockerImageTag	Version of the image (e.g., 0.1.0)	Required
dockerRepository	Full image name (e.g., airbyte/destination-db)	Required
baseImage	The Java or Python base image with digest	Required
releaseStage	Maturity level (alpha, beta, generally_available)	Required

Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:65-100, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:175-185

Docker Build Flow

The project utilizes Gradle plugins to automate the assembly of Docker images. The airbyte-connector-docker-convention plugin reads the metadata and configures the build arguments.

The diagram above illustrates the automated pipeline from source code to a deployable Docker image.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120-160

Runtime Configuration and Deployment

At runtime, connectors are deployed as containers that receive configurations via standard IPC (Inter-Process Communication) and environment variables.

Environment and Data Channels

Airbyte connectors must define their data channel capabilities in the metadata.yaml and application-connector.yml. The platform sets the transport and serialization at runtime.

• Mediums: STDIO or SOCKET
• Formats: JSONL or PROTOBUF
• Serialization: Versioned protocol (e.g., 0.0.2)

Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:100-115, airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:180-210

Resource Management

Dockerized connectors are configured with specific JVM arguments to prevent container crashes and optimize resource usage within the Docker environment. Common defaults include:

• -XX:+ExitOnOutOfMemoryError
• -XX:MaxRAMPercentage=75.0

Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:135-145

Deployment Quality Assurance

Airbyte enforces strict QA checks on Dockerized connectors before they are released to registries like DockerHub. These checks are integrated into the CI/CD pipeline to maintain platform stability.

Critical Deployment Checks

1. Base Image Compliance: Python connectors must not use a custom Dockerfile; they must declare a maintained base image (docker.io/airbyte/python-connector-base) in metadata.yaml.
2. HTTPS Only: All network requests from the container must use HTTPS for security compliance.
3. Metadata Validity: The metadata.yaml must match the actual image properties and follow the Semantic Versioning scheme.
4. Acceptance Tests: High-use connectors must have enabled acceptance test suites within their metadata configuration.

Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:320-350, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:230-245

Docker Component Relationship

The relationship between the platform and the deployed connector is governed by a strict lifecycle of operations.

This sequence shows the interaction between the platform and a connector deployed via Docker.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:215-250, airbyte-integrations/developer-docs/destinations/preflight-checklist.md:15-30

Testing and Local Development

For development, Airbyte provides mechanisms to iterate on Docker images locally before deployment.

• Local Build: Developers can use ./gradlew :destination-{name}:assemble to build local images and docker images to verify creation.
• Testcontainers: The project extensively uses Testcontainers to spin up ephemeral database containers during integration tests, mimicking a real Docker deployment environment.
• Documentation Previews: For documentation changes, local servers can be started via pnpm start (standard Docusaurus) to verify how documentation rendered within the Dockerized UI will appear.

Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:195-210, airbyte-integrations/developer-docs/destinations/preflight-checklist.md:35-45, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md:110-120

Docker deployment is the cornerstone of Airbyte's scalability and connector-first architecture. By standardizing the packaging, metadata, and runtime behavior of containers, Airbyte ensures that hundreds of heterogeneous integrations can be managed, tested, and executed through a unified platform interface.