SDK Testing Framework
Relevant source files
The following files were used as context for generating this wiki page:
The SDK Testing Framework provides a standardized, automated suite of tests for Singer taps and targets developed using the Meltano Singer SDK. It is designed to ensure that custom-built extractors and loaders are compliant with the Singer specification and handle data correctly across various scenarios, such as incremental syncing, schema validation, and batch processing.
The framework integrates directly with pytest via a custom plugin and offers "standard" test suites that can be easily invoked by developers to validate their implementations without writing extensive boilerplate test code.
Sources: CLAUDE.md, pyproject.toml:285-286, singer_sdk/testing/pytest_plugin.py
Framework Architecture
The testing framework is structured around the pytest ecosystem, utilizing a custom entry point named singer_testing. It provides high-level factory functions that generate test classes for specific tap or target instances.
Core Testing Components
| Component | Description |
|---|---|
| Standard Suites | Pre-defined collections of tests for taps and targets. |
| Pytest Plugin | An entry point (singer_testing) that registers the SDK's testing utilities with the pytest runner. |
| Tap Test Factory | Functions like get_standard_tap_tests that return a suite of tests for a given Tap class. |
| Target Test Factory | Functions like get_standard_target_tests that return a suite of tests for a given Target class. |
Sources: singer_sdk/testing/pytest_plugin.py, pyproject.toml:285-286, singer_sdk/testing/__init__.py
Testing Flow
The following diagram illustrates how the SDK testing framework executes a standard suite against a Tap or Target.
The framework automates the invocation of the Tap/Target CLI, processes the resulting Singer messages, and validates them against expected schemas or state behaviors.
Sources: singer_sdk/testing/tap_tests.py, singer_sdk/testing/target_tests.py
Tap Testing
Tap tests focus on verifying the extraction logic, schema discovery, and state management. The framework includes specific checks for stream discovery and record emission.
Standard Tap Tests
The framework includes tests such as:
- Discovery: Ensures the tap can successfully run in discovery mode and output a valid catalog.
- Sync: Verifies that the tap can emit RECORD, SCHEMA, and STATE messages correctly.
- Incremental Sync: Checks if the tap respects bookmarks and correctly handles replication keys.
Sources: singer_sdk/testing/tap_tests.py, singer_sdk/testing/__init__.py:13-16
Sources: singer_sdk/testing/tap_tests.py
Target Testing
Target tests verify the loading capabilities of a plugin, ensuring it correctly processes incoming Singer streams and handles batching or SQL operations where applicable.
Target Validation Scenarios
The framework tests targets by piping sample data into them and observing the results. Key areas include:
- Record Processing: Ensuring the target accepts and processes RECORD messages without error.
- Batching: For targets using
BatchSink, verifying that records are correctly accumulated and flushed. - State Forwarding: Confirming that the target correctly handles and emits STATE messages as required by the Singer spec.
Sources: singer_sdk/testing/target_tests.py, singer_sdk/testing/__init__.py:18-21
Integration and Pytest Plugin
The framework is integrated into the project's build system via pyproject.toml. It defines a pytest11 entry point which allows pytest to automatically load the singer_sdk.testing.pytest_plugin module.
Test Markers
The SDK uses several markers to categorize tests:
contrib: Tests for experimental or optional modules.external: Tests requiring external resources (like live APIs).packages: Tests for the sample taps and targets included in the repository.snapshot: Tests utilizingpytest-snapshotfor output verification.
Sources: pyproject.toml:139-147, pyproject.toml:285-286, CLAUDE.md
Implementation Example
To use the framework, a developer typically imports the factory functions and defines a test class:
from singer_sdk.testing import get_standard_tap_tests
from my_tap.tap import MyTap
def test_standard_tap_tests():
"""Run standard tap tests."""
tests = get_standard_tap_tests(MyTap, config={})
for test in tests:
test()
Sources: singer_sdk/testing/__init__.py, CLAUDE.md
Summary
The SDK Testing Framework is a core utility for Meltano Singer SDK developers, providing automated compliance checks and data validation. By standardizing the testing of Taps and Targets, it reduces the need for manual test implementation while ensuring that plugins adhere to the Singer specification and handle complex data flows like incremental syncing and batch processing reliably.