System Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
- airbyte-integrations/developer-docs/destinations/preflight-checklist.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
- docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/writing-docs.md
Airbyte is an open-source data integration platform designed to move data from various sources to destinations using a modular connector-based architecture. The system prioritizes trust, reliability, and security, operating as an infrastructure component within a user's data pipeline. By adhering to Extract-Load-Transform (ELT) principles, the architecture ensures that data transformations occur primarily at the destination, maintaining flexibility and backward compatibility.
The architecture is built around a Dataflow CDK (Connector Development Kit), which provides a structured framework for building destination connectors. This framework segregates logic into specialized components such as SQL Generators, Database Clients, and Insert Buffers to handle complex database interactions, schema evolution, and high-performance batch writes.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:46-70, airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:143-157
Connector Architecture and Components
The Airbyte architecture relies on a clear separation of concerns between the platform and individual connectors. Destination connectors developed via the CDK follow a specific component-based model to manage the lifecycle of data synchronization.
Core CDK Components
| Component | Responsibility |
|---|---|
| SqlGenerator | Responsible for generating dialect-specific SQL. These are pure functions and highly testable. |
| Client | Handles I/O operations and executes the SQL generated by the SqlGenerator. |
| InsertBuffer | Manages efficient batch writes specific to the target database. |
| StreamLoader | Orchestrates the table lifecycle (e.g., creation, swap, deletion). |
| Writer | The high-level orchestrator that implements minimal logic to coordinate other components. |
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:143-157
Data Flow Lifecycle
The data flow within a destination connector moves from the platform's standard input through a managed lifecycle, ending with state messages sent back to the platform.
This diagram illustrates the internal orchestration of a destination connector during a write operation.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:159-170
Database Management and SQL Generation
Destination connectors must handle diverse database operations including namespace management, table operations, and type mapping. The SqlGenerator is the primary interface for creating these instructions.
Table Operations and Atomic Swaps
Connectors must implement standard operations such as createTable(), dropTable(), tableExists(), and countTable(). A critical architectural feature for reliability is the Atomic Swap, which allows for overwriting data by renaming or exchanging tables (e.g., ALTER TABLE old_table SWAP WITH new_table) to avoid data inconsistency windows.
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:58-85
Schema Evolution Logic
Airbyte supports automatic schema changes through a multi-step process:
- Discover: Introspect the existing schema via system catalogs (e.g.,
information_schema.columns). - Compute: Determine the required schema based on incoming data.
- Compare: Identify differences (new columns, type changes).
- Apply: Execute
ALTER TABLEoperations to ADD, DROP, or MODIFY columns.
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:129-134, airbyte-integrations/developer-docs/destinations/preflight-checklist.md:173-200
Type System Mapping
The Column Utilities component maps Airbyte's internal types to database-specific dialects. Connectors must handle various types including:
- JSON: JSONB, VARIANT, or TEXT fallback.
- Timestamps: Handling precision and timezones (e.g.,
TIMESTAMPTZ). - Metadata: Airbyte-specific columns like
_airbyte_raw_id(UUID),_airbyte_extracted_at, and_airbyte_meta.
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:88-129
Data Synchronization Modes
The architecture supports multiple sync modes, each requiring specific SQL logic implemented in the SqlGenerator and StreamLoader.
Deduplication and Upsert
For "Dedupe" mode, the system utilizes window functions like ROW_NUMBER() OVER to group by primary keys and select the latest records. The actual data merge is handled via MERGE statements or INSERT ON CONFLICT (UPSERT) mechanisms.
This sequence shows the typical flow for deduplicated synchronization using a temporary staging table.
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:148-170, airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:159-170
Documentation and UX Architecture
The Airbyte project uses Docusaurus for its technical documentation, which is integrated into the build process and versioned to match platform releases.
Documentation Stack
- Build Tool: Docusaurus
- Language: Markdown
- Deployment: Vercel
- Diagrams: Mermaid.js for architecture and flow visualization.
- Linting: Vale (style and prose) and MarkDownLint (structure).
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md:7-12, 335-350, docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/writing-docs.md:280-295
Versioning and Multi-Instance Structure
The documentation is split into multiple instances:
- Home: Non-versioned.
- Platform: Versioned (e.g., version-2.0, version-2.1).
- Connectors: Non-versioned.
- Release Notes: Non-versioned.
When a new major version of Airbyte is released, the docs are "frozen" by creating a physical copy of the platform docs in /docusaurus/platform_versioned_docs.