Mappers & Stream Maps
Relevant source files
The following files were used as context for generating this wiki page:
Mappers are a specialized type of plugin in Meltano designed to perform inline data transformations between an extractor (tap) and a loader (target). They implement the "Stream Maps" capability, allowing users to filter streams, alias properties, or create new calculated fields using expressions while data is in transit.
This system is built primarily on the Singer SDK's Stream Maps capability, enabling declarative transformations defined in the meltano.yml project file. Mappers are particularly useful for anonymizing sensitive data, flattening schemas, or performing simple row-level logic without requiring a full heavy-duty transformation layer like dbt.
Sources: src/meltano/schemas/meltano.schema.json, src/meltano/core/plugin/singer/mapper.py
Architecture and Components
The Mapper system is integrated into the Meltano plugin ecosystem as a first-class citizen alongside extractors and loaders. A mapper plugin essentially acts as a Singer target that consumes data, applies transformations, and then acts as a Singer tap to emit the transformed data.
Mapper Plugin Class
The SingerMapper class represents the implementation of mapper plugins within the Meltano core. It inherits from SingerPlugin and provides specific logic for handling mappings defined in the project configuration.
The diagram above shows the relationship between the base plugin structure and the specialized mapper implementation.
Sources: src/meltano/core/plugin/singer/mapper.py:10-21
Mapping Objects
A mapper can contain multiple "mappings," which are invokable named configurations. Each mapping consists of a unique name and a specific configuration dictionary that defines how stream maps should be applied.
Sources: src/meltano/schemas/meltano.schema.json:710-730
Stream Maps Capability
Stream Maps are the engine behind mappers. They allow for row-level transformations using a declarative syntax. Key features include:
- Property Aliasing: Mapping a source property to a new name.
- Filtering: Dropping specific rows based on an expression.
- Calculated Fields: Generating new properties using functions like
md5()or standard Python expressions. - Schema Flattening: Automatically expanding nested properties into a flat structure.
Data Flow with Mappers
When a mapper is included in a meltano run or meltano elt command, it sits in the middle of the data pipeline.
This diagram illustrates how data flows through a mapper plugin during an integration job.
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:1150-1180, src/meltano/schemas/meltano.schema.json:620-650
Configuration and Schema
Mappers are defined in the plugins section of the meltano.yml file. They require a name, executable, and a mappings array.
Configuration Structure
| Field | Type | Description |
|---|---|---|
name | String | Unique identifier for the mapper plugin (e.g., meltano-map-transformer). |
executable | String | The command-line executable name (default: meltano-map-transform). |
mappings | Array | A list of named mapping configurations containing stream_maps. |
capabilities | Array | Usually includes stream-maps and about. |
Sources: src/meltano/schemas/meltano.schema.json:731-755
Example Configuration
Mappers use stream_maps to define transformations per stream. Below is a conceptual representation of a mapping that masks IP addresses.
{
"name": "meltano-map-transformer",
"plugin_type": "mappers",
"mappings": [
{
"name": "hide-ips",
"config": {
"stream_maps": {
"users": {
"ip_address": "md5(record['ip_address'])",
"email": null
}
}
}
}
]
}
In this example, the ip_address field is hashed, and the email field is dropped from the stream (set to null).
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:1165-1175
Core Implementation Details
The SingerMapper class handles the resolution of configuration and environment variables.
Mapping Logic
The mappings property in SingerMapper extracts mapping definitions from the plugin's configuration. If a specific mapping is requested via the environment (using MELTANO_EXTRACT__MAPPING_NAME), the mapper focuses on that specific transformation set.
Sources: src/meltano/core/plugin/singer/mapper.py:23-55
Environment Variables
Meltano automatically generates environment variables for mappers to facilitate plugin execution:
MELTANO_MAP_TRANSFORMER__MAPPING_NAME: The name of the currently active mapping.MELTANO_MAP_TRANSFORMER_STREAM_MAPS: A JSON string of the stream map definitions.
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:1152-1160
Conclusion
Mappers and Stream Maps provide a powerful, code-first approach to data transformation within Meltano pipelines. By leveraging the SingerMapper implementation and declarative stream_maps configurations, developers can efficiently manage data quality and privacy requirements directly within their ELT workflows. The system's ability to sit inline between taps and targets ensures that transformations are applied consistently regardless of the underlying data source or destination.