Singer Specification Concepts
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Singer specification is an open-source standard for data extraction and loading, defining how "Taps" (extractors) and "Targets" (loaders) communicate. In the Meltano Singer SDK, these concepts are implemented through a standardized set of message types, stream definitions, and state management protocols to ensure interoperability between different data tools.
This wiki page outlines the core architectural components of the Singer spec as implemented in the SDK, including message processing, stream mapping, and the lifecycle of data transformation.
Core Message Types
The Singer specification communicates via a sequence of JSON messages. The SDK defines these in the SingerMessageType enumeration.
| Message Type | Description |
|---|---|
RECORD | Contains the actual data items being extracted or loaded. |
SCHEMA | Defines the JSON Schema structure for a specific stream. |
STATE | Tracks the progress of a sync (bookmarks) for incremental extraction. |
ACTIVATE_VERSION | Signals a version change for a specific stream's data. |
BATCH | Defines a manifest for group-based data processing. |
Sources: singer_sdk/singerlib/encoding/base.py:38-44, singer_sdk/plugin_base.py:647-651
Message Processing Flow
The SDK utilizes GenericSingerReader and GenericSingerWriter interfaces to handle the serialization and deserialization of these messages.
The reader processes lines from a file input or standard input, identifies the type key in the JSON line, and executes the corresponding callback function.
Sources: singer_sdk/singerlib/encoding/base.py:53-84
Taps and Streams
A Singer Tap is the source component responsible for extracting data. Data within a tap is organized into "Streams," which represent individual data endpoints (e.g., users, orders).
Stream Metadata
Streams are defined by several key attributes:
- Name: The unique identifier for the stream.
- Schema: A JSON Schema definition describing the record structure.
- Primary Keys: Fields that uniquely identify a record.
- Replication Key: A field (like
updated_at) used to track progress for incremental syncs.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:28-35, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:46-51
Sync Strategies
Taps support different replication methods:
- Full Table: Extracts all data every time.
- Incremental: Uses a
replication_keyandSTATEmessages to only extract records newer than the last sync point.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:144-149
Targets and Sinks
A Singer Target receives Singer messages and loads them into a destination. The SDK uses "Sinks" to handle the implementation logic for different serialization methods.
Serialization Methods
Targets can process data in different ways depending on the destination's capabilities:
| Method | Component | Use Case |
|---|---|---|
| Per Record | process_record() | Real-time APIs, webhooks, or streaming services. |
| Per Batch | process_batch() | Bulk APIs or file uploads (e.g., S3). |
| SQL | SQLSink | Databases (PostgreSQL, MySQL) supporting MERGE/UPSERT. |
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:37-75
Target Data Flow
Inline Stream Mapping
Mappers act as intermediaries that transform data in-flight between a Tap and a Target. This is implemented via the InlineMapper class.
Transformation Capabilities
Mappers can perform several operations without storing state:
- Aliasing: Renaming streams or fields.
- Filtering: Excluding records based on conditions (e.g.,
__filter__: "price > 100"). - Computing: Adding new fields using Python-like expressions (e.g.,
"total": "price * quantity"). - Masking: Hashing or removing sensitive PII data.
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:31-45, singer_sdk/mapper_base.py:31-45
Mapping Architecture
The InlineMapper inherits from both BaseSingerReader and BaseSingerWriter, allowing it to consume a stream, apply map_schema_message or map_record_message logic, and immediately emit the results.
Sources: singer_sdk/mapper_base.py:31-72
Configuration and Validation
All Singer plugins (Taps, Targets, Mappers) use a standardized configuration system based on JSON Schema.
Configuration Syncing
Plugin settings are defined in a config_jsonschema. When used with Meltano, these must stay in sync with the meltano.yml plugin definition and environment variables.
| Setting Component | Source |
|---|---|
config_jsonschema | Defined in the Python class (tap.py or target.py). |
settings block | Defined in meltano.yml. |
.env | Environment variables (e.g., TAP_MYSOURCE_API_KEY). |
Sources: singer_sdk/plugin_base.py:270-295, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:188-220
Config Validation Logic
The SDK automatically validates input configuration against the provided schema using the JSONSchemaValidator. If required fields are missing or types are incorrect, the plugin raises a ConfigValidationError.
Sources: singer_sdk/plugin_base.py:461-490
Summary
Singer Specification Concepts in the Meltano SDK center around a robust message-passing architecture. By standardizing RECORD, SCHEMA, and STATE messages, the SDK enables seamless data movement from Taps to Targets. The inclusion of InlineMapper components allows for complex transformations and data privacy operations to be performed in-memory during the ELT process, while the configuration system ensures that all components remain strictly typed and validated.