Java/Kotlin CDK
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
- airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md
- airbyte-cdk/bulk/CONTRIBUTING.md
- airbyte-integrations/developer-docs/destinations/preflight-checklist.md
- airbyte-integrations/developer-docs/destinations/step-by-step/7-troubleshooting.md
- airbyte-cdk/java/airbyte-cdk/core/src/main/kotlin/io/airbyte/cdk/integrations/BaseConnector.kt
- airbyte-cdk/bulk/core/base/src/main/kotlin/io/airbyte/cdk/command/ConfiguredCatalogFactory.kt
The Java/Kotlin CDK (often referred to as the Kotlin Bulk CDK or Dataflow CDK) is a high-performance framework designed for building Airbyte destination connectors. It leverages Kotlin and the Micronaut framework to provide a structured, dependency-injection-driven environment for handling large-scale data synchronization. The CDK focuses on "Bulk" operations, optimizing the path from the Airbyte platform's record stream to high-throughput database writes.
The framework is built to support incremental development through clear milestones, starting from basic specification handling to complex production features like schema evolution, Change Data Capture (CDC), and deduplication modes. It requires JDK 21 or higher and uses Gradle for build automation and Dokka for documentation generation.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:1-10, airbyte-cdk/bulk/CONTRIBUTING.md:1-10
Core Architecture and Component Roles
The architecture of the Java/Kotlin CDK is centered around a decoupled design where SQL generation is separated from execution, and high-level orchestration is separated from low-level batching.
Component Breakdown
| Component | Responsibility | Technical Nature |
|---|---|---|
| SqlGenerator | Generates SQL strings for table operations, namespaces, and writes. | Pure functions, highly testable. |
| Client | Executes SQL generated by the SqlGenerator. | Handles I/O and database-specific error classification. |
| InsertBuffer | Collects records and performs efficient batch writes. | Database-specific (e.g., Multi-row INSERT or COPY). |
| StreamLoader | Orchestrates the lifecycle of a table (Create, Write, Swap, Drop). | Provided by CDK; selected by developer. |
| Writer | High-level orchestration of the write operation. | Implemented by developer with minimal logic. |
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:105-115, airbyte-integrations/developer-docs/destinations/preflight-checklist.md:5-10
Operational Data Flow
The following diagram illustrates the flow of data from the Airbyte platform through the CDK components into the target database.
This flow ensures that data is aggregated efficiently before being committed to the destination, with the StreamLoader managing the finalization (like atomic swaps) to ensure data integrity.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:118-125
Configuration and Specification
The CDK uses a two-tier system for configuration to separate user-facing UI requirements from internal runtime logic.
Specification vs. Configuration
- Specification (
{DB}Specification): Defines the UI form schema (JSON Schema) that users interact with in the Airbyte UI. It includes annotations for titles, descriptions, and secrets. - Configuration (
{DB}Configuration): A clean Kotlin data class used by the connector code at runtime. - Factory (
{DB}ConfigurationFactory): Acts as the transformation layer, converting the validated Specification POJO into the runtime Configuration object.
Dependency Injection (DI) Contexts
The CDK utilizes Micronaut for dependency injection. A critical aspect of development is understanding the three distinct contexts in which a connector runs, as they have different bean requirements and catalog handling.
DI Runtime Environments
| Context | Purpose | Catalog Handling |
|---|---|---|
| Component Test | Fast unit testing of components. | Uses MockDestinationCatalog; bypasses TableCatalog. |
| Integration Test | Full system validation. | Spawns connector process; parses real JSON catalog. |
| Docker Runtime | Production execution. | Uses real platform catalog; requires application-connector.yml. |
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/7-troubleshooting.md:10-50
Common DI Registration Requirements
To avoid "No bean found" errors, specific components must be registered as @Singleton in a BeanFactory or class definition:
- Name Generators:
FinalTableNameGenerator,RawTableNameGenerator, andColumnNameGeneratorare required forTableCataloginstantiation. - Status Gatherers:
DatabaseInitialStatusGatherermust be provided for the Writer to track sync progress. - Operation Beans:
WriteOperationV2must be declared and marked with@Requires(property = "airbyte.operation", value = "write").
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/7-troubleshooting.md:85-130
Sync Modes and Advanced Features
The Java/Kotlin CDK supports several sync strategies that dictate how data is moved and committed.
Supported Sync Modes
- Append: Data is written directly to the final table.
- Overwrite: Data is written to a temporary table, followed by an atomic swap with the final table.
- Append Dedup: Uses a primary key to merge new data into the existing table, often utilizing window functions like
ROW_NUMBER().
Advanced Capabilities
- Schema Evolution: The CDK provides patterns to discover existing schemas, compute differences with incoming data, and apply
ALTER TABLEstatements automatically. - Generation ID Tracking: Supports tracking sync generations to ensure atomic commits and facilitate cleanups of failed attempts.
- CDC Support: Handles Change Data Capture records, supporting hard and soft deletes based on metadata fields.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:55-75, airbyte-integrations/developer-docs/destinations/preflight-checklist.md:110-150
Project Scaffolding
Connectors built with the Kotlin CDK follow a standardized directory structure and rely on the airbyte-bulk-connector Gradle plugin for dependency management.
Directory Layout
destination-{db}/
├── src/main/kotlin/io/airbyte/integrations/destination/{db}/
│ ├── check/ # Check operation logic
│ ├── client/ # Database client and SQL generation
│ ├── config/ # Name generators and DI config
│ ├── spec/ # Configuration and Specification classes
│ └── write/ # Write operation logic (loaders/transforms)
├── metadata.yaml # Connector metadata and build options
└── build.gradle.kts # Plugin and driver configuration
Build Configuration
The build.gradle.kts file uses the airbyte-bulk-connector plugin, which automatically resolves core artifacts based on a pinned cdkVersion in gradle.properties.
plugins {
id("airbyte-bulk-connector")
}
airbyteBulkConnector {
core = "load"
toolkits = listOf("load-db")
}
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:15-80
The Java/Kotlin CDK provides a robust, type-safe, and high-performance foundation for Airbyte destinations. By enforcing a separation of concerns between SQL generation, command execution, and orchestration, it enables developers to build production-ready connectors that are both maintainable and highly scalable.