Introduction to Meltano Singer SDK
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Singer SDK is a Python framework designed to streamline the development of Singer-compliant data connectors, including taps (extractors), targets (loaders), and mappers (transformers). By providing a set of abstract base classes and built-in utilities, the SDK allows developers to write significantly less code—approximately 70% less on average—while ensuring automatic compliance with the Singer specification, the open-source standard for extract and load pipelines.
Sources: README.md:1-25, CLAUDE.md:1-5
Core Architecture and Components
The SDK utilizes a plugin-based architecture where developers inherit from abstract base classes to implement specific connector logic. This structure ensures a consistent interface across different types of plugins.
Plugin Types
The SDK supports three primary categories of plugins:
| Plugin Type | SDK Base Class | Purpose |
|---|---|---|
| Tap | Tap | Extracts data from a source (API, Database, File). |
| Target | Target | Loads data into a destination. |
| Mapper | InlineMapper | Transforms data in-flight between a tap and a target. |
Sources: CLAUDE.md:112-117, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:32-40
Data Flow Overview
The following diagram illustrates how data flows through an ELT pipeline utilizing SDK-built components:
The diagram shows the logical progression of data from source systems through the modular components of the Singer SDK.
Sources: README.md:41-50, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:15-25, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:15-25
Taps: Data Extraction
Taps are responsible for connecting to source systems and emitting Singer-formatted messages. The SDK provides specialized classes for different source types, such as RESTStream, GraphQLStream, and SQLStream.
Key Extraction Concepts
- Streams: Individual data endpoints (e.g., users or orders).
- Replication Keys: Used for incremental syncs to track progress (e.g.,
updated_at). - Paginators: Built-in classes like
JSONPathPaginatororHeaderLinkPaginatorhandle API pagination automatically. - Authenticators: Implements logic for API Keys, OAuth2, JWT, or Basic Auth.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:30-45, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:85-115
Targets: Data Loading
Targets receive records, schemas, and state messages to load data into a destination. The SDK supports various serialization methods to optimize performance based on the destination type.
Serialization Methods
- Per Record: Immediate processing, ideal for REST APIs or webhooks.
- Per Batch: Groups records (up to
max_size) for bulk uploads, reducing overhead. - SQL: Specialized for databases, offering automatic table creation, schema evolution, and MERGE/UPSERT support.
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:33-85
Mappers: In-flight Transformation
Mappers sit between taps and targets to modify data without requiring a persistent storage layer. They utilize "Stream Maps" to perform inline transformations.
Supported Transformations
- Filtering: Use
__filter__to remove records based on Python-like expressions. - Aliasing: Rename fields or entire streams.
- Computed Fields: Add new fields using expressions (e.g.,
"total": "price * quantity"). - Masking: Hash sensitive PII data using functions like
md5().
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:43-60, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:118-125
Common SDK Features
All SDK plugins benefit from a unified approach to configuration, error handling, and environment management.
Configuration Management
Plugins define their configuration using JSON Schema. The SDK automatically supports environment variable overrides and .env file loading.
| Type | SDK Type Class | Meltano Kind |
|---|---|---|
| String | StringType | string |
| Integer | IntegerType | integer |
| Boolean | BooleanType | boolean |
| Object | ObjectType | object |
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:195-215, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:205-220
Error Handling Hierarchy
The SDK uses a structured exception hierarchy to determine if a failure is recoverable.
This hierarchy allows the SDK to intelligently handle transient network issues versus permanent configuration errors.
Sources: CLAUDE.md:42-55
Conclusion
The Meltano Singer SDK provides a robust, standardized framework for building data connectors. By abstracting the complexities of the Singer specification—such as state management, catalog discovery, and message formatting—it allows developers to focus on the unique logic of their specific source or destination systems while maintaining a high degree of performance and reliability.