Declarative / Low-Code CDK
Relevant source files
The following files were used as context for generating this wiki page:
- docusaurus/platform_versioned_docs/version-1.6/connector-development/config-based/low-code-cdk-overview.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
- airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
- airbyte-integrations/developer-docs/destinations/preflight-checklist.md
The Declarative / Low-Code CDK (Connector Development Kit) is a framework designed to commoditize the creation of REST API source connectors. It allows developers to define connector behavior using a YAML manifest file rather than writing repetitive Python code. This framework is a specialized part of the Python CDK that maps YAML configurations to actual behavior implementations, focusing on common API patterns such as authentication, pagination, and rate limiting.
Core Philosophy and Benefits
The low-code paradigm is based on the observation that API source connectors are formulaic. Most solve variations of the same seven problems:
- Request Orchestration: Managing endpoints and HTTP methods.
- Authentication: Implementing OAuth or API keys.
- Pagination: Handling limit-offset, page-number, cursor, or header link strategies.
- Rate Limiting: Managing backoff strategies (exponential, fixed, or variable).
- Schema Definition: Describing data structures for downstream normalization.
- Data Decoding: Handling formats like JSON, XML, or CSV and compression (GZIP, BZIP).
- Incremental Sync: Tracking state using cursors (usually date-based).
By using off-the-shelf components, developers reduce bugs, decrease development effort, and improve long-term maintainability.
YAML Manifest Structure
The Connector Manifest is a structured YAML file that serves as the blueprint for the connector. It is divided into four primary top-level sections:
| Component | Description |
|---|---|
version | Indicates the framework version (e.g., "0.1.0"). |
definitions | Reusable objects and anchors to keep the YAML DRY (Don't Repeat Yourself). |
streams | The core logic defining individual data resources (analogous to database tables). |
check | Defines how to validate the connection (e.g., attempting a small read). |
spec | Defines the user input fields (credentials, URLs) required to configure the connector. |
Stream Configuration
Each stream requires specific sub-components to handle data extraction:
- Data Retriever: The engine that manages the
Requester(HTTP details),Pagination(navigation logic), andRecord Selector(extracting data from the response). - Incremental Sync: Configuration for checkpointing and cursor fields to track progress between syncs.
- Partition Router: Logic for splitting a stream into sub-tasks (e.g., requesting data per-region or per-account).
Logic Flow for Data Retrieval
The following diagram illustrates how the low-code components interact during a standard sync operation:
The flow ensures that every HTTP response is parsed into records, transformed, and emitted while maintaining state for the next sync.
Sources: docusaurus/platform_versioned_docs/version-1.6/connector-development/config-based/low-code-cdk-overview.md:76-101, airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:144-154
UX and Design Guidelines
Connectors developed via the CDK must adhere to strict UX standards to ensure reliability and ease of use. The UX Handbook outlines a hierarchy of needs for connector development:
- Security: No leaking credentials in logs or communications.
- Data Integrity: Replicated data must be complete and formatted correctly (e.g., maintaining leading zeros in zipcodes).
- Reliability: Connectors should work 99.9% of the time, handling large inputs and esoteric characters.
- Ease of Use: Sensible defaults and clear input documentation.
Input Configuration Standards (spec.json)
- Avoid Implementation Details: Users should not see tuning parameters like "retry count" unless strictly necessary as an "Advanced" option.
- Fast Fail: Use JSON schema patterns to validate inputs before the sync starts.
- Titles & Descriptions: Every parameter must have a clear
titleanddescriptionto assist non-technical personas like Data Analysts.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:46-77, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:162-184
Output Schema Management
Low-code connectors should strongly favor ELT (Extract-Load-Transform) over ETL. Data should be loaded into the destination with the schema as unchanged as possible.
| Requirement | Purpose |
|---|---|
| Automation | Use tools to generate schemas from OpenAPI definitions to ensure reliability. |
| Completeness | Describe schemas in as much detail as possible to support strongly-typed destinations (e.g., BigQuery). |
| Compatibility | Avoid breaking changes to schemas; only modify if the previous schema was incorrect. |
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:252-282
Documentation and Context-Aware Rendering
Documentation for low-code connectors is written in Markdown but uses custom extensions to handle different environments (Cloud vs. Self-Managed).
- Environment Tags: Content can be wrapped in
<!-- env:oss -->or<!-- env:cloud -->to show/hide instructions based on the platform. - Field Anchors: The
<FieldAnchor field="field_name">tag allows the Airbyte UI to automatically scroll to relevant documentation when a user selects a specific configuration field. - PyAirbyte Integration: Connectors with PyAirbyte enabled automatically generate usage examples unless a manual
Usage with PyAirbytesection is provided.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:15-55, docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:95-105
Summary
The Declarative / Low-Code CDK simplifies the Airbyte ecosystem by moving away from imperative code toward declarative configuration. By centralizing common API logic into reusable YAML components, it allows developers to focus on source-specific details while ensuring the resulting connectors meet the project's high standards for reliability, security, and user experience.