Schema Evolution

Relevant source files

The following files were used as context for generating this wiki page:

• airbyte-integrations/developer-docs/destinations/implementation-reference.md
• airbyte-integrations/developer-docs/destinations/step-by-step/5-advanced-features.md
• airbyte-integrations/developer-docs/destinations/preflight-checklist.md
• airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
• docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md

Schema Evolution is the automated process within Airbyte destination connectors that detects and applies changes to the destination database structure when the source schema changes. It ensures that data replication remains continuous and reliable even as upstream data structures evolve, preventing sync failures that would otherwise occur due to schema mismatches. Sources: implementation-reference.md:315, 5-advanced-features.md:12

The system primarily handles operations such as adding new columns, dropping removed columns, and modifying existing column types or constraints. This capability is a core part of the "Production Path" for connector development, enabling enterprise-grade reliability by maintaining data integrity without manual intervention. Sources: step-by-step/0-introduction.md:46, 5-advanced-features.md:17-21

Architecture and Core Components

Schema evolution is orchestrated by the TableSchemaEvolutionClient interface and implemented within the specific database client of a destination connector. It relies on a coordinated flow between discovering the current state of the database and computing the desired state based on the incoming data stream. Sources: implementation-reference.md:40-62, 5-advanced-features.md:144

The Four-Step Evolution Process

The evolution logic follows a strict four-step sequence to ensure accuracy:

1. Discover: The connector queries the destination's system catalog (e.g., information_schema) to determine the current table structure.
2. Compute: The connector determines the "expected" schema by mapping the Airbyte stream definition to database-specific types.
3. Compare: The framework compares the actual and expected schemas to generate a ColumnChangeset.
4. Apply: The connector executes the necessary ALTER TABLE statements to align the destination with the expected schema.

Sources: implementation-reference.md:318-333, 5-advanced-features.md:144-156

Component Interaction Diagram

The following diagram illustrates how different internal components interact during the schema evolution phase of a sync.

The flow starts when a stream is initialized; if the table already exists, the evolution logic verifies the schema before allowing data to be written. Sources: implementation-reference.md:213-228, 5-advanced-features.md:158-167

Implementation Details

Schema Discovery and Computation

To detect changes, the connector must first implement discoverSchema(). This typically involves querying database metadata tables like information_schema.columns or using database-specific commands like DESCRIBE TABLE. Sources: implementation-reference.md:337-345, 5-advanced-features.md:30-65

override suspend fun discoverSchema(tableName: TableName): TableSchema {
    val columns = getColumnsFromDb(tableName)
    return TableSchema(columns)
}

Note: Connectors must filter out Airbyte internal metadata columns (e.g., _airbyte_raw_id) during discovery to avoid treating them as user data. Sources: 5-advanced-features.md:58-64

Table Alteration Logic

The alterTable function in the SqlGenerator is responsible for generating the specific SQL strings for the evolution.

Operation	Complexity	Strategy
Add Column	Low	ALTER TABLE ... ADD COLUMN
Drop Column	Low	ALTER TABLE ... DROP COLUMN
Modify Constraint	Medium	Dropping or adding NOT NULL
Type Change	High	Temp column approach or Table recreation

Sources: implementation-reference.md:353-388, 5-advanced-features.md:86-130

Handling Incompatible Type Changes

When a column type changes in a way that is not "safe" (e.g., narrowing a type or changing a string to an integer), the connector uses a multi-step temporary column strategy to preserve data:

1. Add a temporary column with the new type.
2. Cast data from the original column into the temporary column.
3. Rename the original column to a backup name.
4. Rename the temporary column to the original name.
5. Drop the backup column.

Sources: implementation-reference.md:374-388, 5-advanced-features.md:109-125

UX and Design Principles

Schema evolution must balance technical correctness with user experience. According to the UX Handbook, changes to output schemas are considered "UX-impacting" and must be handled with caution. Sources: ux-handbook.md:38, ux-handbook.md:225

Reliability Hierarchy

The hierarchy of needs for Airbyte users places Data Integrity and Reliability above all else. Schema evolution serves these needs by ensuring that:

• Data is not lost during type changes (Data Integrity).
• Syncs do not crash when fields are added (Reliability).
• Inferred schemas (e.g., from MongoDB) are treated with "best effort" but prioritize correctness over expressiveness.

Sources: ux-handbook.md:76-105, ux-handbook.md:197-205

ELT vs ETL

Airbyte strongly favors ELT (Extract-Load-Transform). Schema evolution follows this by reflecting the source schema in the destination as closely as possible, rather than attempting to transform it into a "cleaner" version during the load process. This ensures backwards compatibility and reduces the development bottleneck on the Airbyte team. Sources: ux-handbook.md:162-185

Summary

Schema evolution is a critical advanced feature that moves a connector from a proof-of-concept to a production-ready state. By implementing the TableSchemaEvolutionClient, developers enable connectors to autonomously manage database migrations. This maintains a seamless data flow for users, adhering to Airbyte's core principles of trust, reliability, and ELT-driven architecture. Sources: step-by-step/0-introduction.md:55, ux-handbook.md:73