SDK Plugin Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The SDK Plugin Architecture provides a Python-based framework for building data connectors that comply with the Singer Spec. It abstracts the complexities of the Singer protocol, allowing developers to focus on the logic of data extraction (Taps), data loading (Targets), or in-flight data transformation (Mappers).
The architecture is built on top of abstract base classes that define the lifecycle and capabilities of different plugin types. By using the SDK, developers can implement about 70% less code compared to manual Singer implementations while gaining automatic support for features like state management, schema validation, and standardized CLI interfaces.
Sources: README.md:1-25, CLAUDE.md:109-115
Core Plugin Types
The SDK categorizes plugins into three primary functional areas: Taps, Targets, and Mappers. Each type utilizes specific base classes to handle the Singer stream of messages (SCHEMA, RECORD, STATE).
Class Hierarchy and Relationships
The following diagram illustrates the relationship between the core SDK classes and their specialized implementations for various data protocols.
The architecture uses abstract base classes for entry points (Tap, Target) and their respective data handlers (Stream, Sink).
Sources: CLAUDE.md:109-115, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:13-20
Plugin Summary Table
| Plugin Type | Base Class | Primary Responsibility | Key Sub-components |
|---|---|---|---|
| Tap | Tap | Extract data from source | Stream, RESTStream, SQLStream |
| Target | Target | Load data into destination | Sink, BatchSink, SQLSink |
| Mapper | InlineMapper | Transform data in-flight | stream_maps, PropertyMapping |
Sources: CLAUDE.md:109-115, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:15-20
Tap Architecture (Data Extraction)
Taps are responsible for connecting to a source system and emitting Singer messages. The SDK provides high-level abstractions for common protocols like REST, GraphQL, and SQL.
Tap Components
- Tap Class: The entry point that defines configuration schemas and discovers available streams.
- Streams: Represent individual data endpoints (e.g., "users" or "orders").
- Client/Authenticator: Handles API communication, session management, and credential exchange.
The Tap lifecycle starts with configuration and stream discovery, followed by authenticated requests and paginated record emission.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:16-30, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:88-100
Pagination Strategies
The SDK provides several built-in paginators to handle different API behaviors:
JSONPathPaginator: For tokens found in the response body.SimpleHeaderPaginator: For APIs using RFC 5988 Link headers.OffsetPaginator/PageNumberPaginator: For numeric-based traversal.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:104-135
Target Architecture (Data Loading)
Targets consume Singer messages from a Tap (or Mapper) and write them to a destination. The SDK supports two main serialization modes: "Per record" and "Per batch."
Sink Processing Flow
Sinks are the primary unit of work within a Target, handling the accumulation and delivery of records.
The Target architecture buffers records in Sinks until a batch size limit (e.g., max_size) is reached, then triggers the loading process.
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:13-35, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:52-70
Serialization Methods
| Method | Implementation | Best Use Case |
|---|---|---|
| Per record | process_record() | REST APIs, Webhooks, streaming services |
| Per batch | process_batch() | Bulk APIs, file uploads (S3, GCS) |
| SQL | SQLSink | Relational databases (PostgreSQL, SQLite) |
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:37-90
Mapper Architecture (Transformation)
Mappers sit between Taps and Targets to transform data in-flight without persisting it to disk. This is implemented via the SDK's inline stream mapping capabilities.
Mapping Capabilities
- Filtering: Using
__filter__to include/exclude records based on expressions. - Aliasing: Renaming streams or fields.
- Cloning: Creating multiple output streams from a single input source.
- Computation: Adding fields via Python-like expressions (e.g.,
"total": "price * quantity").
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:28-55
Mappers apply logic to records in-memory during the pipeline execution.
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:15-20
Configuration and Metadata
All plugins use a standardized configuration system based on JSON Schema.
AboutInfo Data Structure
The SDK provides a standard mechanism for plugins to report their metadata via the --about CLI flag. The AboutInfo dataclass (found in singer_sdk/about.py) tracks the following:
nameandversionsdk_versionsupported_python_versions(currently 3.10-3.14)capabilities(e.g.,catalog,discover,state)settings(Config JSONSchema)
Sources: singer_sdk/about.py:91-103, singer_sdk/about.py:27-28
Exception Handling
The SDK enforces a specific exception hierarchy to manage sync behavior:
FatalSyncError: Aborts the sync (e.g.,FatalAPIError).RetriableSyncError: Triggers backoff/retry logic (e.g.,RetriableAPIError).ConfigValidationError: Raised when user configuration fails validation.
Sources: CLAUDE.md:46-55
Conclusion
The SDK Plugin Architecture provides a structured, class-based approach to building Singer-compliant connectors. By centralizing common logic like pagination, authentication, and batching within base classes like RESTStream and SQLSink, the architecture ensures consistency across the Meltano ecosystem while reducing the maintenance burden for plugin developers.
Sources: README.md:15-30, CLAUDE.md:109-122