Extractors & Loaders
Relevant source files
The following files were used as context for generating this wiki page:
Extractors and Loaders form the core of the Meltano data integration engine. Meltano utilizes a declarative, code-first approach to manage these components, which are primarily based on the Singer.io standard for taps and targets. Extractors (taps) are responsible for pulling data from various sources such as APIs and databases, while Loaders (targets) handle the insertion of that data into a destination system.
The system supports over 600 connectors, enabling users to orchestrate complex ELT (Extract, Load, Transform) pipelines. These plugins are managed within the meltano.yml project configuration and can be extended or customized through environment-specific overrides and configuration settings.
Sources: README.md:16-17, src/meltano/schemas/meltano.schema.json:149-160
Architecture and Data Flow
The architecture follows a standard pipeline where data is streamed from an extractor to a loader. This is often supplemented by mappers that can transform data mid-stream.
High-Level Execution Flow
Meltano invokes the extractor to discover source schemas or extract data based on a provided catalog. The extracted data is then piped into a loader, which may utilize batching or specific database dialects for efficient insertion.
The diagram shows the standard flow of data from a source system through the Meltano-managed plugins to a destination.
Sources: README.md:1-5, integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json:34-118
Pipeline Components
Meltano defines specific capabilities and metadata for both extractors and loaders to manage their behavior during execution.
| Component | Role | Key Capabilities |
|---|---|---|
| Extractor | Pulls data from source | discover, catalog, state, properties, stream-maps |
| Loader | Pushes data to destination | batch, soft-delete, hard-delete, structured-logging |
Sources: src/meltano/schemas/meltano.schema.json:442-457, src/meltano/schemas/meltano.schema.json:473-485
Extractor Configuration (Extractors)
Extractors are defined as plugins with unique namespaces and executable names. They support sophisticated selection rules and metadata overrides.
Selection and Filtering
Extractors use a select array to determine which entities and attributes to retrieve. Patterns like *.* are used to select all data, while negative patterns (e.g., !traffic_*.*) exclude specific streams.
The flowchart illustrates how Meltano processes selection and filter rules to generate the final data catalog used during extraction.
Sources: src/meltano/schemas/meltano.schema.json:416-431, integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json:206-210
Extractor Specific Settings
| Setting | Type | Description |
|---|---|---|
catalog | string | Path to a catalog file for sync mode. |
load_schema | string | The target database schema for extracted data. |
metadata | object | Overrides for Singer stream and property metadata. |
use_cached_catalog | boolean | Determines if the catalog cache should be used. |
log_parser | string | Specifies the parser (e.g., singer-sdk) for structured logs. |
Sources: src/meltano/schemas/meltano.schema.json:387-414
Loader Configuration (Loaders)
Loaders receive Singer streams and persist them. They often require specific database dialect definitions to inform downstream transformers (like dbt) about the destination type.
Dialect and Compatibility
The dialect property identifies the type of the target database (e.g., snowflake, target_yaml). This allows Meltano to manage environment variables like MELTANO_LOAD__DIALECT which are injected into the plugin's runtime.
Example: Snowflake Loader
A loader like target-snowflake includes complex settings for batch management and data integrity:
- Batch Size:
batch_size_rows(default 100,000) controls the frequency of load operations. - Data Flattening:
data_flattening_max_levelmanages nested JSON properties. - Integrity:
primary_key_requiredcan stop loads if no primary key is defined to prevent duplicates.
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json:1344-1500, src/meltano/schemas/meltano.schema.json:488-491
Shared Plugin Properties
Both Extractors and Loaders inherit from a generic plugin definition, providing common management features.
Common Metadata Fields
- name: The unique identifier for the plugin.
- variant: The specific distribution/author of the plugin.
- pip_url: The installation source (PyPI, Git URL, or local path).
- executable: The command used to run the plugin.
- python: The specific Python version required for the plugin's virtual environment.
Sources: src/meltano/schemas/meltano.schema.json:211-300
Settings and Environment Variables
Meltano dynamically generates environment variables for plugin configuration. A setting named api_key for a plugin named tap-slack will be available as TAP_SLACK_API_KEY and MELTANO_EXTRACT_API_KEY.
This class diagram shows the inheritance relationship where specific plugin types extend the base plugin properties.
Sources: src/meltano/schemas/meltano.schema.json:206-495, AGENTS.md:28-30
Virtual Environment Management
Each extractor and loader is installed into its own isolated virtual environment. Meltano supports multiple backends for managing these environments, specifically virtualenv and uv. The system allows overriding the Python executable per plugin if it requires a different version than the Meltano core.
Sources: src/meltano/schemas/meltano.schema.json:96-107, AGENTS.md:11-13
Conclusion
Extractors and Loaders are the functional units of Meltano's data movement. By standardizing on the Singer protocol, Meltano provides a unified interface for configuring, installing, and running diverse connectors while maintaining isolation through per-plugin virtual environments and flexible configuration via meltano.yml.