Meltano Configuration Sync

Relevant source files

The following files were used as context for generating this wiki page:

• cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md
• cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md
• cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/meltano.yml
• cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/meltano.yml
• README.md

Meltano Configuration Sync refers to the critical process of maintaining alignment between a Singer plugin's internal configuration schema (defined in Python code via the SDK) and the external project configuration defined in meltano.yml. This synchronization ensures that settings discovered by Meltano are correctly validated, typed, and passed to the underlying plugin during execution.

Within the Meltano ecosystem, configuration exists in three primary locations: the plugin's source code (using config_jsonschema), the Meltano project file (meltano.yml), and environment variables (typically documented in .env.example). Failure to keep these in sync leads to configuration drift, resulting in runtime errors or ignored settings.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:204-209, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:286-291, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:195-200

Synchronization Architecture

The architecture of configuration sync relies on mapping high-level Meltano "kinds" to specific Python data types defined using the Singer SDK's typing (th) module. The SDK uses these definitions to generate a JSON Schema that Meltano uses for discovery and validation.

Synchronization Flow

The following diagram illustrates how configuration flows from Meltano into the SDK-based plugin:

This flow ensures that any setting defined in the Meltano project matches the expected schema in the plugin's implementation.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:211-218, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:293-300

Type Mapping Requirements

To maintain sync, developers must map Python types from singer_sdk.typing to the corresponding Meltano kind in meltano.yml.

Python SDK Type	Meltano Kind
StringType	string
IntegerType	integer
BooleanType	boolean
NumberType	number
DateTimeType	date_iso8601
ArrayType	array
ObjectType	object

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:235-243, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:317-325

Implementation Components

1. Internal Schema Definition

Plugins define their configuration using a config_jsonschema property in the Tap, Target, or Mapper class. This uses the PropertiesList utility to define fields, types, defaults, and security constraints.

# Example Internal Definition
config_jsonschema = th.PropertiesList(
    th.Property("api_url", th.StringType, required=True),
    th.Property("api_key", th.StringType, required=True, secret=True),
    th.Property("batch_size", th.IntegerType, default=100),
).to_dict()

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:214-219, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:296-301

2. Meltano Plugin Definition

The meltano.yml file must contain a matching settings block. Properties marked as secret=True in the SDK must be marked with sensitive: true in Meltano.

# Corresponding meltano.yml block
plugins:
  extractors:
    - name: tap-example
      settings:
        - name: api_url
          kind: string
        - name: api_key
          kind: string
          sensitive: true
        - name: batch_size
          kind: integer
          value: 100

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/meltano.yml:23-45, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:221-233

3. Environment Variable Sync

For security and CI/CD, settings are mirrored as environment variables. The naming convention typically follows the plugin name and setting name in uppercase.

# .env.example
TAP_EXAMPLE_API_URL=https://api.example.com
TAP_EXAMPLE_API_KEY=your_api_key_here
TAP_EXAMPLE_BATCH_SIZE=100

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:224-226, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:306-308

Synchronization Best Practices

To avoid configuration drift, the following practices are recommended:

• Atomic Updates: Always update tap.py/target.py/mapper.py, meltano.yml, and .env.example in the same commit.
• Consistent Defaults: Ensure default values defined in the Python Property match the value provided in the Meltano settings block.
• Shared Descriptions: Keep descriptions consistent between the Python code docstrings and the description fields in meltano.yml.
• Validation Groups: Use settings_group_validation in meltano.yml to enforce required setting combinations discovered from the SDK's required=True flags.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:245-248, cookiecutter/tap-template/{{cookiecutter.tap_id}}/meltano.yml:54-61, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:327-330

Conclusion

Meltano Configuration Sync is a foundational requirement for building reliable Singer plugins with the Meltano SDK. By strictly adhering to the mapping between Python types and Meltano settings, developers ensure their plugins are easily discoverable and correctly configured within the Meltano orchestration environment.