ELT Pipelines
Relevant source files
The following files were used as context for generating this wiki page:
Meltano is a declarative, code-first data integration engine designed to build and run ELT (Extract, Load, Transform) pipelines. It replaces the need for custom-coded API integrations by leveraging a plugin-based architecture, primarily following the Singer specification for data movement.
The core purpose of ELT pipelines in Meltano is to move data from a source (Extractor) to a destination (Loader), potentially applying transformations (Mappers or Transformers) during or after the process. This workflow is managed via a central configuration file, meltano.yml, and executed through a command-line interface.
Sources: README.md:1-15, src/meltano/schemas/meltano.schema.json:7-10
Pipeline Architecture
Meltano pipelines are composed of modular components called plugins. These plugins are categorized by their role in the data lifecycle. The system is designed to be declarative, meaning users define the desired state and components in configuration, and Meltano handles the execution and environment management.
Component Roles
| Component Type | Description |
|---|---|
| Extractor | Plugins (taps) that fetch data from APIs or databases. |
| Loader | Plugins (targets) that write data to a destination. |
| Mapper | Plugins that perform inline data mapping/transformation between extraction and loading. |
| Transformer | Plugins (like dbt) that transform data within the destination after loading. |
| Orchestrator | Tools (like Airflow) that schedule and manage the execution of pipelines. |
Sources: src/meltano/schemas/meltano.schema.json:140-185, README.md:65-70
Data Flow Execution
The data flow typically begins with an extractor producing a stream of data in the Singer format (JSON-based). This stream is piped into a loader. If mappers are present, they intercept the stream to modify schemas or data before it reaches the loader.
This diagram illustrates the standard path of data from extraction through loading and subsequent transformation.
Sources: integration/example-library/meltano-run/index.md:25-85, src/meltano/schemas/meltano.schema.json:335-360
Pipeline Configuration
Pipelines are configured within the meltano.yml file. This includes plugin definitions, environment-specific overrides, and job definitions.
The Job Entity
A job in Meltano is a named set of tasks that can be referenced for execution. Tasks within a job usually represent a sequence of plugins to run together, such as an extractor followed by a loader.
{
"jobs": [
{
"name": "gitlab-to-jsonl",
"tasks": [
"tap-gitlab target-jsonl"
]
}
]
}
Sources: src/meltano/schemas/meltano.schema.json:690-725, integration/example-library/meltano-run/index.md:38-45
Environments
Meltano supports multiple environments (e.g., dev, staging, prod), allowing pipelines to use different configurations (like database credentials) based on where they are running.
The configuration used by the pipeline is determined at runtime by the selected environment.
Sources: src/meltano/schemas/meltano.schema.json:100-110, src/meltano/schemas/meltano.schema.json:727-810
Pipeline Execution Commands
Meltano provides two primary commands for running pipelines: meltano run and meltano elt.
meltano run
The meltano run command is the modern standard for executing blocks of plugins. It can handle multiple blocks in a single invocation and manages state automatically.
- Usage:
meltano run <extractor> <loader> <transformer> - Job Reference:
meltano run <job_name>
Sources: integration/example-library/meltano-run/index.md:30-50, integration/example-library/meltano-basics/index.md:25-30
Configuration Management
Settings can be adjusted dynamically via the CLI, which updates the environment or the meltano.yml file. Sensitive information is typically stored in a .env file rather than the main configuration.
meltano config <plugin_name> set <setting_name> <value>
Sources: integration/example-library/meltano-basics/index.md:15-22, integration/example-library/meltano-run/index.md:15-25
State and Incremental Syncs
Meltano pipelines support incremental replication through state management. The state backend (defaulting to a system SQLite database) stores bookmarks provided by extractors. When a pipeline runs again, the extractor receives the previous state and only fetches new or updated records.
| Setting | Type | Description |
|---|---|---|
state_backend.uri | string | URI for the state storage (default: systemdb). |
state_backend.lock_timeout_seconds | integer | Duration a state lock is valid. |
auto_install | boolean | Automatically install missing plugins on execution. |
Sources: src/meltano/schemas/meltano.schema.json:54-95, integration/meltano-manifest/expected-manifests/meltano-manifest.json:20-30
Summary
ELT Pipelines in Meltano are declarative workflows constructed from interchangeable plugins. By defining extractors, loaders, and transformers in meltano.yml, developers can create scalable data integration logic that is version-controlled and environment-aware. The system handles the complexities of stream management, state persistence, and plugin installation, allowing users to focus on data logic.