Connector Testing Strategies
Relevant source files
The following files were used as context for generating this wiki page:
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md
- docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md
- docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md
- docusaurus/platform_versioned_docs/version-1.6/connector-development/ux-handbook.md
Connector testing strategies in Airbyte focus on ensuring high reliability, security, and data integrity for all integration components. The strategy involves a combination of automated static analysis (QA checks), functional acceptance tests, and manual verification to meet infrastructure-level standards. Because Airbyte acts as an infrastructure detail for moving data, connectors are expected to work with at least 99.9% reliability, handling esoteric characters, various timezones, and large-scale inputs without failure.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:46-77, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:12-20
Automated QA Checks and Static Analysis
Airbyte employs a suite of automated QA checks within the CI/CD pipeline to enforce standards across documentation, metadata, packaging, and security. These checks act as a set of guidelines that a connector must follow to be released to registries like DockerHub or PyPi.
Documentation and Metadata Validation
Connectors are validated for the presence and structure of critical files such as metadata.yaml and icon.svg. Documentation is checked for a specific header structure, valid links, and the existence of migration guides for breaking changes.
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:22-160
Packaging and Security Requirements
Automated tests verify that connectors adhere to specific technical requirements:
- Dependency Management: Python connectors must use Poetry.
- Licensing: Must be licensed under MIT or Elv2.
- Security: Connectors must use HTTPS only for external requests and use the standard Airbyte Python base image.
- Versioning: Must follow Semantic Versioning.
Acceptance Testing
Acceptance tests are required for all medium to high-use source connectors. These tests ensure the connector behaves as expected when interacting with actual or simulated data sources.
Test Configuration Logic
The following diagram illustrates the flow of automated checks that validate a connector's readiness for integration.
The diagram shows the sequential requirement from static analysis through metadata validation and final acceptance testing before a connector is released.
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/resources/qa-checks.md:12-280
User Experience (UX) and Reliability Testing
Testing strategies are not limited to code execution but also encompass the "predictability" of the connector. Reliability is the top concern in the hierarchy of user needs, followed by security and data integrity.
Reliability Hierarchy
Testing must confirm that the connector handles:
- Confidential Data: No leaking of credentials in logs.
- Data Integrity: Correct handling of zip codes (leading zeros), timezones, and UTF characters.
- Edge Cases: Large inputs and weirdly formatted data.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:65-98
Persona-Based Testing
Tests are designed to ensure usability for different technical personas:
| Persona | Technical Level | Testing Focus |
|---|---|---|
| Data Analyst | Low | Setup guide clarity, UI tooltips, and default configurations. |
| Analytics Engineer | Medium | SQL/DBT compatibility and Git-based workflows. |
| Data Engineer | High | API performance, Docker/K8s infrastructure, and CLI interactions. |
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/ux-handbook.md:122-140
Documentation and UI Verification
Documentation is considered a part of the connector's functional surface area and must be tested for accuracy across different environments (Airbyte Cloud vs. Open Source).
Environment-Specific Rendering
Airbyte uses custom Markdown extensions to test and show/hide content based on the environment:
<HideInUI>: Content hidden in-app but visible on the docs site.<!-- env:oss -->: Content specific to self-managed versions.<!-- env:cloud -->: Content specific to Airbyte Cloud.
Sources: docusaurus/platform_versioned_docs/version-2.1/connector-development/writing-connector-docs.md:15-68
Field Anchors and UI Mapping
Documentation is mapped to specific UI components using the <FieldAnchor> tag. This allows the in-app help panel to automatically scroll to the relevant instructions when a user selects a configuration field. This is tested using jsonpath expressions.
<FieldAnchor field="replication_method.replication_slot">
...instructions...
</FieldAnchor>
Local Testing Environment
Developers are encouraged to test connectors locally before submission.
Local Development Workflow
- Platform Testing: Run a local Airbyte build to view rendered output of connector documentation.
- Documentation Build: Use Docusaurus locally (
pnpm start) to verify Markdown rendering and Mermaid diagrams. - Linting: Use Vale (for style/grammar) and MarkDownLint (for structure) to ensure documentation meets quality standards.
Sources: docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md:100-145, docusaurus/platform_versioned_docs/version-2.0/contributing-to-airbyte/writing-docs.md:325-450
Conclusion
Connector testing at Airbyte is a multi-layered approach that prioritizes infrastructure-grade reliability. By combining strict static analysis (QA checks) with functional acceptance tests and persona-focused UX guidelines, the project ensures that data movement remains secure, accurate, and easy to configure for users ranging from analysts to data engineers. Documenting and verifying environment-specific behaviors through custom Markdown extensions further ensures that the user experience is consistent across Cloud and Open Source platforms.