Python CDK

Relevant source files

The following files were used as context for generating this wiki page:

• docusaurus/platform_versioned_docs/version-2.1/connector-development/cdk-python/README.md
• docusaurus/platform_versioned_docs/version-2.0/connector-development/cdk-python/README.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
• airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md

The Airbyte Python Connector Development Kit (CDK) is a framework designed for the rapid development of production-grade Airbyte connectors. It provides a structured approach to building source connectors by offering specialized helpers for HTTP APIs (REST, GraphQL, etc.) and generic Python sources. The framework abstracts the complexities of the Airbyte Protocol, allowing developers to focus on the specific logic of the data source.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/cdk-python/README.md:10-15, docusaurus/platform_versioned_docs/version-2.0/connector-development/cdk-python/README.md:10-15

Core Architecture and Concepts

The Python CDK is built around the concept of "Streams," which represent the different tables or resources available from a source. Developers primarily interact with the CDK by implementing specific classes and functions that define how to discover, check, and read data from these streams.

Source Implementation Components

The CDK architecture requires the implementation of several key components to create a functional Airbyte Source.

Component	Description
HTTP Streams	Specialized helpers for interacting with REST or GraphQL APIs.
Full Refresh Streams	Streams that retrieve all available data during every sync.
Incremental Streams	Streams that only retrieve new or modified data since the last sync using a cursor.
Connector Builder	A low-code IDE powered by the Python CDK for rapid prototyping.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/cdk-python/README.md:10-33, docusaurus/platform_versioned_docs/version-2.0/connector-development/cdk-python/README.md:10-33

Stream Logic Flow

The following diagram illustrates the high-level logic flow within a CDK-based source connector during a sync operation.

Stream Types and Implementation

Full Refresh Streams

Full refresh streams are the simplest implementation, requiring the definition of a path for the API endpoint and a parse_response function to extract records from the API's JSON output.

Incremental Streams

Incremental syncs optimize data transfer by tracking the state of the sync. Key requirements for these streams include:

• Cursor Field: The field in the data (e.g., updated_at) used to track progress.
• Stream Slices: A method (stream_slices) to break down a large data request into smaller, manageable chunks based on the cursor.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/cdk-python/README.md:33-40, docusaurus/platform_versioned_docs/version-2.0/connector-development/cdk-python/README.md:33-40

Development and UX Standards

Connectors developed with the Python CDK are expected to adhere to specific User Experience (UX) and technical standards to ensure reliability and trust.

Guiding Principles

• Trust & Reliability: Connectors must work 99.9% of the time, ensuring data integrity (e.g., preserving leading zeros in zip codes, correct timezone handling).
• ELT over ETL: The CDK encourages Extract-Load-Transform. Data should be loaded into the destination as unchanged as possible to ensure backward compatibility and flexibility.
• Fail Fast & Actionable: Connection checks and configuration validation should provide clear, precise feedback to the user.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:40-100, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:180-210

Configuration (spec.json)

The spec.json defines the input parameters required from the user. Best practices include:

• Adding title and description to every parameter.
• Hiding niche parameters under an "Advanced" section.
• Setting sensible defaults to minimize user friction.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:120-170

Integration with PyAirbyte

The Python CDK integrates with PyAirbyte, allowing connectors to be run within Python scripts. Documentation for these connectors can be automatically generated from the JSON schema spec.

Documentation Components for PyAirbyte

If a connector is PyAirbyte-enabled, developers can manually specify usage sections using the following components:

• <PyAirbyteExample>: Generates a runnable code example.
• <SpecSchema>: Generates a reference table of the connector's JSON schema.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:110-135

Documentation and Single-Sourcing

Connector documentation must support multiple contexts (Cloud, Self-Managed UI, and the public docs site). The project uses custom Markdown tags to manage this:

• <HideInUI>: Content visible only on the documentation site.
• <!-- env:oss -->: Content specific to self-managed Airbyte.
• <!-- env:cloud -->: Content specific to Airbyte Cloud.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:15-60

Conclusion

The Python CDK provides a robust, standardized framework for building Airbyte connectors. By prioritizing the ELT pattern, providing specialized stream helpers, and enforcing strict UX guidelines, the CDK ensures that connectors are reliable, scalable, and easy for end-users to configure across various Airbyte environments.