Plugin Architecture & Lifecycle

Relevant source files

The following files were used as context for generating this wiki page:

• src/meltano/core/plugin/base.py
• src/meltano/core/plugin/project_plugin.py
• src/meltano/core/project_plugins_service.py
• src/meltano/core/plugin/file.py
• src/meltano/core/meltano_file.py
• src/meltano/schemas/meltano.schema.json

Meltano's plugin architecture is a declarative, hierarchical system designed to manage the discovery, installation, and execution of third-party tools within a data project. The system separates the static definition of a plugin (capabilities, variants, and settings) from its project-specific configuration and inheritance. Plugins are categorized by their role in the ELT lifecycle, such as extractors, loaders, and mappers, and are managed through a centralized service layer that handles their state in the project's configuration file.

The lifecycle of a plugin spans from discovery and addition to the project via meltano.yml, through installation into isolated virtual environments, and finally to execution via an invoker. This architecture ensures that different tools with conflicting dependencies can coexist within the same project while maintaining a consistent configuration interface.

Plugin Types and Definitions

Plugins are classified into specific types that define their behavior and expected capabilities within the project. Each plugin type corresponds to a specific stage or utility in the data integration workflow.

Plugin Type Enumeration

The PluginType class defines the supported categories of plugins.

Plugin Type	Descriptor	Primary Verb	Description
EXTRACTORS	extractor	extract	Tools for pulling data from sources (e.g., Singer taps).
LOADERS	loader	load	Tools for pushing data to destinations (e.g., Singer targets).
TRANSFORMS	transform	transform	dbt-style data transformations.
TRANSFORMERS	transformer	transform	Tools that run transformations (e.g., dbt).
ORCHESTRATORS	orchestrator	orchestrate	Workflow management tools (e.g., Airflow).
FILES	file bundle	-	Plugins that manage project file templates.
UTILITIES	utility	utilize	General purpose CLI tools.
MAPPERS	mapper	map	Tools for inline stream transformations.

Sources: src/meltano/core/plugin/base.py:75-156, src/meltano/schemas/meltano.schema.json:442-451

Data Hierarchy

Meltano uses a three-tier structure to define plugins:

1. PluginDefinition: The base definition containing a namespace and available variants.
2. Variant: A specific version or implementation of a plugin (e.g., different maintainers for the same tap).
3. ProjectPlugin: The project-specific instance that inherits from a definition/variant and holds user configuration.

Sources: src/meltano/core/plugin/base.py:276-318, src/meltano/core/plugin/project_plugin.py:77-133

The diagram above shows the relationship between plugin references, definitions, variants, and project-specific plugins. Sources: src/meltano/core/plugin/base.py:276, src/meltano/core/plugin/project_plugin.py:77

Plugin Inheritance and Discovery

Plugins in Meltano support deep inheritance, allowing a ProjectPlugin to derive its properties from another plugin or a base definition. This is managed by the ProjectPluginsService.

Inheritance Logic

A plugin can inherit from another named plugin using the inherit_from attribute. If a plugin shadows a base definition with the same name, it inherits from that base definition automatically. The ProjectPluginsService resolves these parents at runtime to provide a complete view of the plugin's capabilities.

• Cyclic Inheritance: The system prevents plugins from inheriting from themselves through a cycle via CyclicInheritanceError.
• Fallback Mechanism: Attributes like pip_url, executable, and capabilities fall back to the parent if not explicitly defined in the child project plugin.

Sources: src/meltano/core/plugin/project_plugin.py:46-74, 163-178, src/meltano/core/project_plugins_service.py:535-555

Source Preference

The ProjectPluginsService can prioritize different sources for plugin definitions using DefinitionSource flags:

• CUSTOM: Embedded custom definitions in meltano.yml.
• LOCKFILE: Definitions stored in local lockfiles to ensure reproducibility.
• INHERITED: Properties derived from other plugins in the project.

Sources: src/meltano/core/project_plugins_service.py:38-46, 520-561

Configuration and Settings

Plugin configuration is split into two categories: config (standard plugin settings) and extras (Meltano-specific internal settings prefixed with _).

Settings Resolution

The ProjectPlugin merges settings from multiple sources:

1. Definitions: Default settings defined in the base plugin variant.
2. Project Level: Values set in meltano.yml under the plugin's config block.
3. Environment Variables: Overrides provided via the system environment, prefixed according to the plugin's name and type (e.g., MELTANO_EXTRACT_...).

Sources: src/meltano/core/plugin/project_plugin.py:270-285, src/meltano/core/project_plugins_service.py:385-400

Extras vs. Config

• Config: Direct parameters passed to the plugin executable (e.g., api_key, start_date).
• Extras: Meltano-specific controls like _select (for extractors), _update (for file bundles), or _mapping_name (for mappers).

Sources: src/meltano/core/plugin/project_plugin.py:255-268, src/meltano/core/plugin/file.py:35-43

Plugin Lifecycle: Add and Install

The lifecycle of a plugin begins when it is added to the project and ends with its execution.

Installation Flow

Plugins (except for Mappers) usually require installation into a virtual environment (venv). The FilePlugin type specifically handles "file bundles" that populate the project root with templates.

This flow illustrates the transition from adding a plugin to the project configuration to its physical installation. Sources: src/meltano/core/project_plugins_service.py:102-154, src/meltano/core/plugin/file.py:218-251

Compatibility Checks

Before execution, Meltano validates that the plugin is compatible with the running Meltano version using the requires_meltano attribute. If the current version does not satisfy the plugin's specifier, an IncompatibleMeltanoVersionError is raised.

Sources: src/meltano/core/plugin/project_plugin.py:417-425, src/meltano/core/meltano_file.py:75-85

Specialized Plugin Behaviors

File Bundle Plugins

FilePlugin (type FILES) manages files within the Meltano project rather than executing a transformation. It provides hooks to create or update project files (like .gitignore or dbt profiles) based on contents bundled within its pip package.

• Header Management: Files managed by these plugins receive a standard header warning users that manual changes may be overwritten.
• Update Logic: The _update extra configures which specific paths should be kept in sync during meltano upgrade.

Sources: src/meltano/core/plugin/file.py:33-100, 218-251

Mapper Plugins

Mappers (type MAPPERS) are unique because a single mapper definition in meltano.yml can generate multiple "mapping" plugins. Each entry in the mappings array of a mapper is treated as a separate invokable plugin instance.

• Virtual Plugins: These generated mappings have _mapping: true in their extra config and are not physically installed.

Sources: src/meltano/core/meltano_file.py:126-140, 192-214, src/meltano/core/plugin/project_plugin.py:400-415

Summary

Meltano's plugin architecture provides a robust framework for managing the lifecycle of diverse data tools. By utilizing a hierarchical inheritance model and strict separation between plugin definitions and project configurations, Meltano ensures that plugins are reproducible, configurable, and isolated. The system's ability to handle complex requirements—from standard ELT tools to file-based templates and dynamic mappers—allows it to serve as a comprehensive control plane for data integration projects.

Sources: src/meltano/core/plugin/base.py, src/meltano/core/project_plugins_service.py, src/meltano/core/plugin/project_plugin.py