Connector Development Kit (CDK) Overview
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
The Connector Development Kit (CDK) is a framework designed to simplify the creation and maintenance of Airbyte connectors. It provides structured patterns, common components, and scaffolding tools that allow developers to build both source and destination connectors efficiently. The CDK ensures that connectors adhere to Airbyte's standards for reliability, security, and data integrity by providing standardized interfaces for common operations like discovery, check, and write.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:1-10, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:14-25
Architecture and Components
The CDK architecture is built around modular components that separate concerns between business logic, data transformation, and physical I/O. For destination connectors, this is particularly evident in the separation of SQL generation from execution.
Destination Architecture Pattern
The dataflow within a destination connector follows a structured lifecycle involving several key roles:
| Component | Description |
|---|---|
| SqlGenerator | Pure functions that generate SQL strings; highly testable and database-agnostic in logic. |
| Client | Executes generated SQL, handles I/O, and manages database-specific error handling. |
| InsertBuffer | Manages efficient batching of records for database-specific ingestion. |
| StreamLoader | Orchestrates the table lifecycle (creation, deletion, swapping). |
| Writer | The high-level orchestrator that implements the business logic for the sync operation. |
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:145-155
Data Flow Diagram
The following diagram illustrates the lifecycle of data moving from the Airbyte platform through the CDK components to the final database destination.
This diagram shows how the platform interacts with the connector through standard streams, triggering a sequence of internal component activations to perform a sync.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:157-168
Core Connector Operations
The CDK enforces the Airbyte protocol by requiring connectors to implement specific operations defined in the connector specification (spec.json or spec.yaml).
Standard Operations
- Spec: Returns the connector capabilities and configuration requirements.
- Check: Validates the provided configuration and tests connectivity to the source or destination.
- Discover: (Sources only) Identifies available streams and their schemas.
- Write: (Destinations only) Handles the ingestion of records into the target database.
Sync Modes
Connectors built with the CDK support various sync modes to handle data updates:
- Append: Direct inserts into the target table.
- Overwrite: Atomic swap using temporary tables to ensure consistency.
- Dedupe: Uses Primary Keys and MERGE/UPSERT logic to prevent duplicate records.
- CDC Support: Handles hard and soft deletes via Change Data Capture logs.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:18-80, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:43-55
Connector UX and Configuration
The CDK promotes a "Security and Reliability First" approach. Configuration via spec.json is the primary way users interact with the connector.
Configuration Guidelines
- Avoid exposure of implementation details: Configurations should focus on what to achieve, not how to do it (e.g., avoid exposing retry counts unless necessary).
- Default Values: Minimize user friction by providing sensible defaults for 95% of use cases.
- Advanced Sections: Technical or niche parameters (like TLS certificates) should be hidden in an "Advanced" section to reduce cognitive load for non-technical personas.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:113-146
Target Personas
The CDK supports different developer and user personas:
| Persona | Technical Level | Focus |
|---|---|---|
| Data Analyst | Proficient in SQL/Excel | Ease of use, clear documentation, and UI guidance. |
| Analytics Engineer | Proficient in SQL, dbt, Python | Reliable schemas and transformation capabilities. |
| Data Engineer | Proficient in multiple languages/infra | Deep technical controls and reliability. |
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:95-108
Quality Assurance and Testing
Connectors developed with the CDK must pass a rigorous set of QA checks to be integrated into the Airbyte platform.
QA Check Categories
- Documentation: Must include a
README.md, user-facing docs under./docs/integrations/, and migration guides for breaking changes. - Metadata: Every connector requires a valid
metadata.yamlat the root, including tags for language and CDK type (e.g.,cdk:low-code,cdk:python). - Packaging: Python connectors must use Poetry for dependency management and follow Semantic Versioning.
- Security: Must use HTTPS for external requests and use the official Airbyte base images.
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:20-350
Testing Strategy
The CDK encourages a three-tiered testing approach:
- Component Tests: Validate individual operations like table creation or SQL generation.
- Integration Tests: Validate the write initialization and full connector lifecycle.
- Basic Functionality Tests: End-to-end validation of all features and sync modes.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:170-175
Documentation and Single-Sourcing
Connector documentation uses custom Markdown extensions to support different environments (Cloud vs. Open Source) from a single source.
Custom Markdown Tags
<HideInUI>: Content visible on the documentation site but hidden inside the Airbyte application.<!-- env:oss -->: Content specific to the Open Source / Self-Managed version.<!-- env:cloud -->: Content specific to the Airbyte Cloud version.<FieldAnchor>: Links a piece of documentation to a specific field in the UI usingjsonpath.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:12-90
Summary
The Connector Development Kit (CDK) serves as the architectural foundation for the Airbyte ecosystem. By enforcing structured data flows, standardizing operations like --spec and --check, and providing a robust set of QA guidelines, it enables the creation of high-quality, reliable, and secure data connectors. Its focus on modularity between SQL generation and execution, combined with comprehensive testing requirements, ensures that connectors remain maintainable across various database environments and user personas.