Singer Protocol Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Singer Protocol Integration in Meltano provides a declarative framework for running Data Extraction and Loading (ELT) operations based on the Singer.io specification. Meltano acts as an orchestrator that manages the configuration, execution, and state persistence of Singer Taps (extractors), Targets (loaders), and Mappers.
By implementing the Singer protocol, Meltano ensures compatibility with a vast ecosystem of open-source connectors. The integration handles the heavy lifting of environment variable expansion, configuration file generation, and incremental state management, allowing developers to focus on data movement rather than plumbing.
Sources: src/meltano/core/plugin/singer/base.py, src/meltano/schemas/meltano.schema.json
Plugin Architecture
Meltano categorizes Singer-compliant executables into three primary plugin types: Extractors, Mappers, and Loaders. All these types inherit from a common base implementation that standardizes how configuration is processed and how subprocesses are invoked.
Core Components
| Component | Description |
|---|---|
SingerPlugin | The base class providing shared logic for configuration processing and environment management. |
SingerMapper | Handles stream transformations using the "Stream Maps" capability. |
SingerTarget | Manages data loading and captures state messages for persistence. |
BookmarkWriter | A specialized handler that parses JSON state messages from a target's stdout. |
Sources: src/meltano/core/plugin/singer/base.py:53, src/meltano/core/plugin/singer/mapper.py:31, src/meltano/core/plugin/singer/target.py:118
Class Hierarchy
The following diagram illustrates the inheritance structure of the Singer integration classes.
Configuration Management
The integration translates Meltano's declarative meltano.yml settings into the JSON configuration files expected by Singer taps and targets.
Processing Logic
Meltano performs several steps before passing configuration to a Singer executable:
- Filtering: Removes null values from the configuration dictionary.
- Nesting: Converts flat keys (e.g.,
db.host) into nested JSON objects. - Environment Expansion: Replaces variables like
${VAR_NAME}within the configuration with values from the project's.envor system environment.
Sources: src/meltano/core/plugin/singer/base.py:78-95, src/meltano/core/plugin/singer/mapper.py:80-87
Configuration Hooks
Meltano uses asynchronous hooks to manage the lifecycle of configuration files:
before_configure: Generates the temporary JSON config file and writes the processed payload.before_cleanup: Deletes the temporary configuration file after execution to prevent sensitive data leakage.
Sources: src/meltano/core/plugin/singer/base.py:113-128
Logging and Observability
Meltano provides advanced logging for Singer plugins, supporting both standard text output and structured logging for SDK-based connectors.
Meltano automatically generates logging configuration files for the subprocesses. For SDK-based plugins, it configures a StructuredFormatter if the _log_parser setting is set to singer-sdk.
Sources: src/meltano/core/plugin/singer/base.py:144-180, CHANGELOG.md:158-161
State Management and Bookmarking
Incremental replication is achieved by capturing STATE messages emitted by Singer Targets. Meltano employs a BookmarkWriter to intercept these messages.
Bookmark Flow
- The Singer Target writes data to a destination.
- Upon successful persistence, the Target emits a JSON
STATEmessage to itsstdout. - The
BookmarkWriter(attached to thePluginInvokeroutput handler) reads this line. - The message is validated and saved to the State Service, which persists it to the configured backend (e.g., System DB, S3, or GCS).
Sources: src/meltano/core/plugin/singer/target.py:53-115, src/meltano/schemas/meltano.schema.json:59-106
Target Invocation Sequence
The following diagram shows how Meltano attaches the state handler during target invocation.
Stream Mapping
The SingerMapper allows for inline transformations of Singer streams. It supports multiple mappings per plugin, which can be selected at runtime.
Mapper Configuration
Mappers use a specific configuration structure defined in the project schema:
_mappings: An array of objects, each containing anameand aconfigblock._mapping_name: The name of the specific mapping to apply during a particular run.
Meltano extracts the specific config for the active mapping and merges it with the base plugin configuration before execution.
Sources: src/meltano/core/plugin/singer/mapper.py:35-51, src/meltano/schemas/meltano.schema.json:497-526
Summary
The Singer Protocol Integration allows Meltano to function as a robust engine for any Singer-compatible connector. It abstracts the complexities of the protocol—such as managing state files, generating complex JSON configurations, and parsing structured logs—into a declarative workflow. Key classes like SingerPlugin and BookmarkWriter ensure that data integrity is maintained through rigorous configuration processing and reliable state persistence across various storage backends.