API & SaaS Source Connectors
Relevant source files
The following files were used as context for generating this wiki page:
- airbyte-integrations/connectors/source-github/source_github/source.py
- airbyte-integrations/connectors/source-stripe/source_stripe/source.py
- airbyte-integrations/connectors/source-slack/AGENTS.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
API & SaaS Source Connectors are critical infrastructure components within the Airbyte data pipeline. Their primary purpose is to move data from external software-as-a-service (SaaS) platforms and APIs into various destinations while maintaining high reliability and trust. These connectors act as an implementation detail in the user's broader goal of data replication, functioning similarly to infrastructure tools like Terraform or Docker.
The scope of these connectors includes managing input configurations (spec.json), ensuring data integrity during extraction, and providing structured output schemas for strongly typed destinations. Design and development are guided by the principle of scalability and high-quality user experience (UX) to serve various technical personas, from Data Analysts to Data Engineers.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:21-36
Core Architecture and Principles
The architecture of API and SaaS connectors is built on a hierarchy of needs that prioritizes trust and reliability over feature set or speed. This ensures that infrastructure remains a time-saver rather than a debugging burden for users.
Hierarchy of Needs
The following diagram illustrates the priority of concerns when developing or maintaining a source connector:
This diagram shows the dependency of user trust on fundamental security and data integrity before higher-level features can be effectively utilized.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:65-70
ELT vs ETL Logic
Connectors strongly favor the ELT (Extract-Load-Transform) pattern over ETL. This approach keeps the transformation responsibility with the consumer at the destination, offering several technical advantages:
- Backwards Compatibility: Maintaining the raw API structure ensures that downstream flows do not break when transformation logic changes.
- Efficiency: Changes to transformation requirements do not require a full re-extraction of data from the source API.
- Flexibility: Avoiding opinionated schemas allows multiple user personas to adapt the data to their specific needs.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:204-235
Component Design
Input Configuration (spec.json)
The spec.json defines how a user interacts with the connector. Developers must adhere to strict UX guidelines to minimize friction:
- Default Values: Minimize required fields by providing sensible defaults for 95% of use cases.
- Advanced Sections: Technical parameters (e.g., TLS certificates, retry counts) must be hidden in an "Advanced" section to avoid confusing non-technical users.
- Actionable Failures: Use JSON-schema "pattern" keywords to validate inputs and provide precise error messages if configuration is invalid.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:144-184
Output Schemas
Connectors must describe output schemas as completely as possible to support strongly typed destinations like Postgres or BigQuery. Reliability is prioritized over expressiveness; an incorrect schema is considered worse than no schema because it causes downstream failures. Automated schema generation from OpenAPI definitions is the recommended standard.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:237-257
Quality Assurance and Standards
Every connector undergoes static analysis and automated checks to ensure it meets Airbyte's standards for security, packaging, and documentation.
| Check Category | Requirement | Description |
|---|---|---|
| Security | HTTPS Only | All requests to external services must use HTTPS. |
| Packaging | Poetry | Python connectors must use Poetry for dependency management and locking. |
| Licensing | MIT / Elv2 | Connectors must be licensed under permissive licenses (MIT or Elv2). |
| Testing | Acceptance Tests | Medium to High use connectors must enable automated acceptance tests in metadata.yaml. |
| Versioning | Semantic Versioning | Versions must follow the Semantic Versioning scheme (Major.Minor.Patch). |
Data Flow for API Connectors
The following sequence diagram represents the typical interaction during a connector sync:
This diagram shows the standard Airbyte protocol steps (Check, Discover, Read) used by API source connectors to extract and load data.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:40-60, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:210-220
Documentation Requirements
Technical documentation is mandatory and must follow a standardized structure to support different personas.
Required Documentation Headers
User-facing documentation must include:
- Prerequisites: Details on required fields and credentials (API keys, tokens).
- Setup Guide: Step-by-step instructions for both Airbyte Cloud and Open Source.
- Supported Sync Modes: Explicit list of supported modes (Full Refresh, Incremental).
- Changelog: A complete record of version changes and migration guides for breaking changes.
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:40-100
Target Personas
Connectors are designed with three primary personas in mind, each with differing technical requirements:
| Persona | Knowledge Level | Connector Focus |
|---|---|---|
| Data Analyst | Excel, SQL, Dashboards | Ease of use, clear UI titles/descriptions, zero-config. |
| Analytics Engineer | SQL, dbt, Git, Python | ELT predictability, documentation on lookback windows. |
| Data Engineer | Docker, K8s, APIs, Orchestration | Reliability, performance, advanced tuning parameters. |
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:107-130
Conclusion
API & SaaS Source Connectors are built on the foundational principles of trust, reliability, and ELT design. By prioritizing security and data integrity over features, and adhering to strict QA checks—such as HTTPS enforcement and Poetry-based packaging—these connectors provide a robust infrastructure for data replication. Their success depends on clear communication through standardized documentation and a configuration design that serves technical and non-technical personas alike.