Sync Modes

Relevant source files

The following files were used as context for generating this wiki page:

• docusaurus/platform_versioned_docs/version-1.8/connector-development/ux-handbook.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
• docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
• docusaurus/platform_versioned_docs/version-1.7/contributing-to-airbyte/writing-docs.md

Sync modes define the logic and methodology used by Airbyte connectors to replicate data from a source to a destination. These modes determine how data is read (e.g., full refresh vs. incremental) and how it is written to the destination (e.g., overwrite vs. append). The selection of a sync mode impacts data integrity, reliability, and synchronization speed, which are the top concerns for Airbyte users.

Within the Airbyte architecture, sync modes are fundamental to the Extract-Load-Transform (ELT) philosophy. By favoring ELT, Airbyte ensures that data is moved with minimal transformation, preserving the original schema as much as possible to maintain backwards compatibility and flexibility for downstream consumers.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:68-81, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:162-175

Core Principles of Data Syncing

Airbyte's approach to syncing data is built upon a hierarchy of user needs, prioritizing security, data integrity, and reliability over speed and ease of use.

Trust and Reliability

Reliability is a critical requirement for sync modes. A connector must handle large inputs and esoteric data types (such as specific UTF characters or timezone-sensitive timestamps) without corruption or data loss. Sync modes like Change Data Capture (CDC) are specifically designed to provide high reliability for database replication by processing logical replication slots.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:79-105, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:275-279

ELT vs. ETL Implementation

Airbyte strongly favors the ELT (Extract-Load-Transform) pattern. Sync modes are designed to extract and load data into a destination while leaving its format largely unchanged. This approach removes Airbyte as a development bottleneck and ensures that the "T" (Transformation) can change without requiring a full re-extraction of data.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:162-187

Sync Mode Types and Behaviors

While specific code-level sync mode definitions (like ConfiguredSyncMode) are typically managed within the CDK, the conceptual behavior is documented through the UX guidelines for connector development.

Change Data Capture (CDC)

CDC is a specialized sync mode used for database replication. It allows Airbyte to track changes in the source database by reading logical replication slots or binary logs.

• Prerequisites: Users must often create logical replication slots and specific permissions in the source database.
• Limitations: Certain character sets (e.g., UTF-16) may not be supported depending on the underlying replication tool (e.g., Debezium).
  Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:275-285, docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:117-142

Incremental Syncing

Incremental syncs rely on cursor fields to only replicate data that has changed since the last sync. This is essential for maintaining sync speed and efficiency. In some APIs (like Facebook Ads), data integrity requires a "lookback window" because data can be updated for up to 28 days after creation.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:107-110, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:143-150

Configuration Elements for Syncing

Parameter Category	Description	UX Requirement
Input Spec	Defined in spec.json	Must include "title" and "description" for every parameter.
Advanced Config	TLS certificates, retry logic	Should be hidden under an "Advanced" section to reduce friction.
Lookback Window	Timeframe for attributing updates	Must clearly document how different values impact data output.
CDC Settings	Replication slots, permissions	Must provide how-tos and clear documentation of assumptions.

Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:126-150, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:264-279

Schema Management in Sync Modes

The way a sync mode interacts with the destination schema is vital for strongly typed destinations like Postgres, BigQuery, or Avro.

Output Schema Reliability

Connectors must describe their output schemas as completely as possible. If a schema cannot be inferred reliably (e.g., in schemaless sources like MongoDB), it is preferred to provide a less expressive but more reliable schema rather than an incorrect one that leads to downstream failures.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:195-212

Backward Compatibility

Breaking changes to output schemas are considered a "necessary evil" and should only occur if the previous schema was incorrect or if the connector is being updated to follow ELT principles.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:214-224

Documentation and User Interface

Sync modes are presented differently depending on the environment (Cloud vs. Self-Managed). Documentation must account for these variations.

Context-Specific Documentation

Airbyte uses custom Markdown tags to tailor sync mode instructions:

• <HideInUI>: Prevents content from appearing in the Airbyte application while keeping it on the docs site.
• <!-- env:oss --> and <!-- env:cloud -->: Single-sourcing tags that show/hide content based on the user's environment.
  Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:15-50

Field Anchors

To assist users in configuring complex sync parameters, documentation can use <FieldAnchor> tags. These link specific fields in the spec.json (such as replication_method) to relevant documentation sections, which automatically highlight when the user interacts with the UI component.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:92-120

Summary

Sync modes in Airbyte represent the technical implementation of reliable data movement. By balancing reliability with the ELT philosophy and providing clear, context-aware documentation, Airbyte enables various personas—from Data Analysts to Data Engineers—to manage complex data pipelines effectively. The focus remains on trust, ensuring that data integrity is maintained regardless of the chosen synchronization methodology.