Meltano Plugin Discovery
Relevant source files
The following files were used as context for generating this wiki page:
Meltano Plugin Discovery is a core service module within the FLEXT Meltano ecosystem designed to query, identify, and retrieve metadata for Meltano plugins (Extractors and Loaders) from the Meltano Hub and active projects. It provides a standardized interface for the system to understand available plugin variants, capabilities, and configurations without requiring manual specification.
The discovery system bridges high-level service requests with the Meltano SDK and Hub API, ensuring that plugin information such as logo_url, description, and available variants are normalized into a consistent format for the FLEXT platform.
Architecture and Core Logic
The discovery process is primarily managed by the FlextMeltanoPluginDiscoveryMixin, which orchestrates the retrieval of plugin data. It leverages a "Temporary Project" pattern to perform discovery operations when an active project context is not provided by the caller.
Discovery Workflow
When a discovery request is initiated, the system follows a structured path to fetch and normalize data:
Key Components
| Component | Responsibility |
|---|---|
FlextMeltanoPluginDiscoveryMixin | Main service mixin providing discover_plugins and fetch_plugin_info methods. |
FlextMeltanoAbstractions | Handles the low-level fetching of plugin types (extractors/loaders) from the Meltano environment. |
FlextMeltanoProjectService | Used to create ephemeral projects required for isolated discovery operations. |
FlextMeltanoUtilitiesRuntime | Provides normalization logic for building canonical discovery payloads. |
Sources: src/flext_meltano/services/meltano_plugin_discovery.py:13-16, src/flext_meltano/_utilities/runtime.py:202-218
Data Models
The discovery system uses Pydantic-based models to ensure type safety and consistent structure for plugin metadata.
Plugin Discovery Models
| Model | Field | Type | Description |
|---|---|---|---|
PluginDiscoverySource | default_variant | str | The recommended variant for the plugin. |
variants | t.JsonMapping | Raw dictionary of available variants. | |
logo_url | str | URL to the plugin icon/logo. | |
PluginDiscoveryItem | name | NonEmptyStr | Unique identifier of the plugin. |
type | NonEmptyStr | Plugin category (e.g., extractor, loader). | |
variants | str | Comma-separated list of variant names. |
Sources: src/flext_meltano/_models/discovery.py:13-44
Data Flow for Metadata Normalization
The system converts raw Meltano metadata into a flattened structure suitable for the FLEXT API.
Plugin Extraction and Filtering
The discovery service often handles large volumes of metadata from the Meltano Hub. To maintain performance, specific limits and filtering logic are applied:
- Limits: By default, the service limits broad discovery to the first 10 extractors and 5 loaders when performing a general scan.
- Type Normalization: Plugin groups (e.g.,
extractors) are mapped to public labels (e.g.,extractor) using theFlextMeltanoUtilitiesRuntime.normalize_discovered_plugin_typehelper. - Variant Handling: Variants are extracted from the plugin definition and joined into a comma-separated string for simplified consumption.
Sources: src/flext_meltano/services/meltano_plugin_discovery.py:84-96, src/flext_meltano/_utilities/runtime.py:277-296
Project-Based Discovery
Discovery can also be performed against the plugins already defined within a Meltano project using the FlextMeltanoProjectManager.
# Extracting plugins via the Project SDK
def _extract_sdk_plugins(self, plugin_type: str | None) -> t.SequenceOf[t.JsonMapping]:
sdk_plugins_service = self._sdk_project.plugins
for plugin in sdk_plugins_service.plugins():
# Filtering logic for plugin_type
# Extraction of name, type, and variant
Sources: src/flext_meltano/services/meltano_project_sdk.py:84-118
Constraints and Constants
The discovery system relies on several fixed configuration values defined in the base constants:
- Plugin Name Lengths: Sources must be at least 5 characters; Targets must be at least 8 characters.
- Timeouts: Plugin discovery operations and installations default to a 300-second timeout.
- Prefixes: The system uses
dbtandtargetprefixes to identify specific plugin categories during normalization.
Sources: src/flext_meltano/_constants/base.py:91-101
Conclusion
Meltano Plugin Discovery provides a robust mechanism for the FLEXT platform to dynamically interact with the Meltano ecosystem. By utilizing temporary project contexts and strict Pydantic models, it ensures that plugin metadata is always available, accurate, and correctly formatted regardless of whether a user has pre-configured a project.