Meltano Hub Discovery
Relevant source files
The following files were used as context for generating this wiki page:
Meltano Hub Discovery is the mechanism used to search, list, and retrieve plugin definitions from Meltano Hub, which serves as the single source of truth for Meltano plugins, Singer taps, and Singer targets. This system allows users to find and integrate over 600+ APIs and databases into their Meltano projects programmatically or via the Command Line Interface (CLI).
Sources: README.md:95-99, src/meltano/core/hub/client.py:1-15
Architecture and Components
The discovery system is built around a service-oriented architecture where a dedicated client manages all interactions with the Hub's REST API.
MeltanoHubService
The MeltanoHubService class is the central component for interacting with the Meltano Hub. It handles the resolution of the Hub's API root URL and manages the underlying HTTP client (HubClient). Recent updates have also embedded this client directly into the lock service to streamline plugin locking.
Key responsibilities include:
- Plugin Retrieval: Fetching all plugins or finding a specific plugin by type and name.
- API Management: Handling authentication and session persistence for Hub requests.
Sources: src/meltano/core/hub/client.py, CHANGELOG.md:275-276
HubClient
The HubClient is a specialized HTTP client that inherits from MeltanoSession. It is responsible for making authenticated requests to the Hub API using a configurable Authorization header.
Sources: src/meltano/core/hub/client.py
The following flowchart illustrates the initialization and discovery flow within the service:
The diagram shows the logic flow from configuration retrieval to the final parsing of plugin definitions from the Hub API.
Configuration
Hub Discovery behavior is controlled through top-level Meltano settings defined in the project schema. These settings determine which Hub instance is queried and how the client authenticates.
| Setting | Type | Default | Description |
|---|---|---|---|
hub_url | string | https://hub.meltano.com | The base URL for the Meltano Hub website. |
hub_api_root | string | null | null | The URL to the root of the Meltano Hub API (e.g., https://hub.meltano.com/meltano/api/v1). |
hub_url_auth | string | bool | null | null | The value for the Authorization header. Supports env var expansion (e.g., Bearer $ACCESS_TOKEN). |
Sources: src/meltano/schemas/meltano.schema.json:237-268, integration/meltano-manifest/expected-manifests/meltano-manifest.json:41
CLI Interface
The Meltano CLI provides a hub command group to allow users to discover plugins directly from the terminal.
Commands
meltano hub list
Lists all plugins available on the Hub for a specific plugin type (e.g., extractors, loaders).
meltano hub search
Searches for plugins on the Hub based on a provided term. This command queries the Hub API and filters results matching the user's input.
Sources: src/meltano/cli/hub.py
The following sequence diagram demonstrates the interaction between the CLI, the Hub Service, and the remote API:
The sequence diagram details the request-response cycle initiated by a CLI user searching for a plugin.
Plugin Discovery and Locking
Meltano utilizes discovery data not only for listing but also for Plugin Locking. When a plugin is added to a project, Meltano can fetch the latest definition from the Hub and "lock" it in a meltano.lock file or merge it into the project manifest. This ensures that the plugin version and its capabilities are preserved for consistent execution across different environments.
Sources: CHANGELOG.md:162-163, src/meltano/core/manifest/init.py:22-23
Summary
Meltano Hub Discovery facilitates the "declarative, code-first" nature of the platform by providing a robust programmatic interface to the plugin ecosystem. By centralizing plugin definitions in MeltanoHubService and providing accessible CLI commands, Meltano ensures that users can discover and pin integrations with minimal manual configuration.