Data Lake & Object Storage Destinations
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/connectors/destination-s3/src/main/kotlin/S3V2Destination.kt
- airbyte-integrations/connectors/destination-gcs/src/main/kotlin/io/airbyte/integrations/destination/gcs_v2/GcsV2Destination.kt
- airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md
- airbyte-integrations/developer-docs/destinations/preflight-checklist.md
- airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md
Data Lake and Object Storage destinations in Airbyte are specialized connectors designed to move data from various sources into large-scale storage systems such as Amazon S3 or Google Cloud Storage (GCS). Unlike traditional relational database destinations, these connectors utilize a "File Transfer" mechanism rather than direct SQL execution for the primary data load, often acting as a staging area or a permanent repository for raw and structured data. Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120, S3V2Destination.kt, GcsV2Destination.kt
Architecture and Components
The architecture for object storage destinations follows the Bulk Cloud Data Kit (CDK) pattern, which separates the lifecycle of a sync into specific operational phases. While database destinations focus on SqlGenerator and TableOperationsClient, data lake destinations leverage a specific file-transfer capability within the CDK core. Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:113, airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120
Core Component Roles
The following diagram illustrates the high-level data flow from the Airbyte platform into an object storage destination.
The flow highlights the transition from the platform's standard input to the internal writer orchestration and eventually to the object storage service. Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:122-132
| Component | Role in Data Lake Destinations |
|---|---|
| Writer | High-level orchestration that manages the stream loader lifecycle. |
| StreamLoader | Handles the actual movement of data to the destination. |
| File Transfer | A configuration flag in application-connector.yml that enables cloud storage optimizations. |
| InsertBuffer | Collects and batches records before they are uploaded as files. |
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:115-120, airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120
Implementation Details for S3 and GCS
The implementations for S3 (S3V2Destination) and GCS (GcsV2Destination) utilize Kotlin and the Airbyte CDK to define the entry point for the connector. These classes are responsible for initializing the AirbyteDestinationRunner. Sources: S3V2Destination.kt:1-9, GcsV2Destination.kt:1-12
Main Entry Point (S3)
package io.airbyte.integrations.destination.s3
import io.airbyte.cdk.AirbyteDestinationRunner
fun main(args: Array<String>) {
AirbyteDestinationRunner.run(*args)
}
Sources: S3V2Destination.kt:5-9
Main Entry Point (GCS)
package io.airbyte.integrations.destination.gcs_v2
import io.airbyte.cdk.AirbyteDestinationRunner
fun main(args: Array<String>) {
AirbyteDestinationRunner.run(*args)
}
Sources: GcsV2Destination.kt:8-12
Configuration and Specification
Object storage destinations require specific configurations to handle authentication, bucket locations, and data formats. This is managed through the Specification and Configuration classes.
Required Connector Properties
For data lake destinations, the application-connector.yml file must explicitly set the file-transfer property to true. This informs the CDK core to use file-based logic instead of relational database logic. Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120
airbyte:
destination:
core:
file-transfer:
enabled: true
Sources: airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:115-120
Key Specification Fields
Common fields for object storage destinations include:
- Authentication: Access Keys for S3 or Service Account JSON for GCS.
- Location: Bucket name and region.
- Pathing: Prefix or folder structure for the stored files.
- Format: Output format such as Parquet, Avro, or JSONL.
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:13-35, airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:155-175
Data Transfer Strategy
Object storage destinations prioritize bulk upload over row-by-row insertion. The "Batch Insert Strategy" involves accumulating data in an InsertBuffer and then flushing it to a file. Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:132-152
This sequence shows the buffering mechanism that ensures efficient network utilization by sending large files instead of individual records. Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:134-145, airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:122-132
Bulk Upload Considerations
- Compression: Support for GZIP, LZ4, or Snappy to reduce storage costs and transfer time.
- Batch Size: Optimal batching (e.g., 10,000 records or 100MB) is critical for performance.
- Staging: Data lakes often serve as staging areas for other destinations (like Snowflake or Redshift) using a
COPYcommand.
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:144-158
Type Mapping and Metadata
Data stored in object storage must be mapped into the destination's type system. For data lakes, this often involves mapping to formats like Parquet or JSON. Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:95-130
| Airbyte Type | Object Storage Representation |
|---|---|
| String | VARCHAR or STRING in Parquet schema. |
| Timestamp | Millisecond precision with timezone. |
| JSON | Native JSONB or STRING depending on file format. |
| Metadata | _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta. |
Sources: airbyte-integrations/developer-docs/destinations/preflight-checklist.md:100-125
Summary
Data Lake & Object Storage destinations leverage the Airbyte Bulk CDK to provide high-throughput data replication into cloud buckets. By enabling the file-transfer mode and implementing robust InsertBuffer and StreamLoader logic, these connectors support massive data volumes with efficient batching and compression. This architecture is fundamental for modern data platforms that separate storage from compute, allowing raw data to be preserved and processed later in tools like Spark, Snowflake, or BigQuery. Sources: airbyte-integrations/developer-docs/destinations/step-by-step/0-introduction.md:113-132, airbyte-integrations/developer-docs/destinations/step-by-step/1-getting-started.md:120