Airbyte Message Protocol
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/developer-docs/destinations/preflight-checklist.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/resources/qa-checks.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
- docusaurus/platform_versioned_docs/version-1.8/connector-development/ux-handbook.md
The Airbyte Message Protocol serves as the foundational communication standard between sources, destinations, and the Airbyte platform. It is designed to facilitate high-reliability data replication through an Extract-Load-Transform (ELT) architecture, prioritizing data integrity and minimal transformation during the movement process.
The protocol ensures that data replicated from diverse sources remains consistent, correct, and complete. By defining standardized metadata columns and a robust type system, it allows destinations to interpret and store records accurately across various database dialects and storage systems.
Core ELT Philosophy
Airbyte strongly favors ELT (Extract-Load-Transform) over traditional ETL. This design choice ensures that data is loaded into the destination as closely as possible to its original format at the source.
Principles of Data Replication
- Backward Compatibility: By avoiding complex transformations during extraction, Airbyte leverages the stability of source APIs, which typically maintain strong versioning guarantees.
- Flexibility: Different user personas (Data Analysts, Analytics Engineers, Data Engineers) have varying needs. Keeping data in its raw form allows each persona to apply their own transformations downstream.
- Efficiency: ETL requires re-extracting data if transformation logic changes. ELT avoids this by moving the transformation responsibility to the destination.
- Data Integrity: The protocol prevents loss of precision (e.g., leading zeros in zipcodes) and handles esoteric characters (UTF characters) to maintain complete accuracy.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:144-180, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:86-100
Data Flow Diagram
The following diagram illustrates the ELT data flow governed by the Airbyte Message Protocol.
Standard Metadata Columns
To ensure auditability and system-level tracking, the protocol requires specific internal metadata columns to be present in every destination table.
| Column Name | Data Type | Purpose |
|---|---|---|
_airbyte_raw_id | UUID (or VARCHAR) | A unique identifier for the specific record version. |
_airbyte_extracted_at | Timestamp | The exact time the record was extracted from the source. |
_airbyte_meta | JSONB / JSON | System-level metadata and diagnostic information. |
_airbyte_generation_id | BIGINT | Tracks the specific generation or sync attempt for the data. |
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:126-140
Type Mapping and Dialects
The protocol defines a universal type system that must be mapped to destination-specific dialects. This mapping ensures that complex types like JSON and Timestamps are handled consistently regardless of the underlying database.
Mapping Requirements
- JSON Storage: Native JSON/JSONB types are preferred. If unavailable, the protocol defaults to TEXT with JSON string encoding.
- Timestamp Handling: Timestamps must support timezone information and provide millisecond precision where possible.
- Boolean Logic: Maps to native BOOLEAN types or fallbacks like TINYINT(1) or BIT.
- Namespace Qualification: Data is structured within namespaces (schemas/databases) depending on the destination's capability.
Schema Evolution and Versioning
Changes to the message protocol or specific connector schemas are strictly managed to prevent downstream pipeline failures.
Breaking Change Management
- Migration Guides: Any breaking change must be accompanied by a migration guide stored in the project documentation.
- Semantic Versioning: Connectors must follow the Semantic Versioning scheme to indicate the nature of changes (major, minor, patch).
- Automated Generation: To ensure reliability, schema descriptions should be automated from OpenAPI definitions or source metadata whenever possible.
Sources: docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/resources/qa-checks.md:21-26, docusaurus/platform_versioned_docs/version-1.6/contributing-to-airbyte/resources/qa-checks.md:255-260, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:195-205
Schema Consistency Sequence
This diagram shows how schemas are validated during the connection process.
Protocol Summary
The Airbyte Message Protocol acts as a contract that guarantees "Infrastructure-as-a-Service" reliability for data pipelines. By enforcing ELT standards and standardized metadata, it ensures that data integrity is maintained at a 99.9% reliability threshold. It eliminates the need for manual transformation during transit, thereby reducing technical debt and enabling flexible downstream analysis.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:47-60, docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:105-115