Introduction to Meltano
Relevant source files
The following files were used as context for generating this wiki page:
Meltano is an open-source, declarative, code-first data integration engine designed to build, run, and orchestrate ELT (Extract, Load, Transform) pipelines. It serves as a single platform to manage the full lifecycle of data integration by leveraging the Singer standard for taps and targets, while providing a unified CLI and configuration layer for orchestration and transformation tools.
Sources: README.md:21-23, src/meltano/schemas/meltano.schema.json:5-6
The platform is built to eliminate the need for maintaining custom API integrations by providing access to over 600+ APIs and databases. Its architecture is centered around specialized service classes and a plugin-based system that allows for highly modular and portable data environments.
Sources: README.md:24-25, AGENTS.md:43-45
Core Concepts and Architecture
Meltano operates as a service-oriented platform where core functionality is implemented through specialized service classes (e.g., SettingsService, ProjectDirsService, VirtualEnvService).
Sources: AGENTS.md:43-45, CHANGELOG.md:43-48
Plugin System
The system is built on various types of plugins, each serving a specific role in the data lifecycle. These are defined in the meltano.yml project file.
| Plugin Type | Purpose | Example / Description |
|---|---|---|
| Extractors | Taps that fetch data from sources. | tap-gitlab, tap-github |
| Loaders | Targets that load data into destinations. | target-postgres, target-jsonl |
| Mappers | Inline stream transformations. | meltano-map-transformer |
| Transformers | Run transformations within the destination. | dbt |
| Orchestrators | Manage job scheduling and execution. | airflow |
| Utilities | General purpose tools for the project. | sqlfluff, great_expectations |
Sources: src/meltano/schemas/meltano.schema.json:104-145, README.md:73-75
Data Flow Architecture
The following diagram illustrates the high-level data flow within a Meltano project, from source extraction to final transformation.
Meltano facilitates the communication between Singer Taps and Targets, managing the state, catalog, and configuration expansion at runtime.
Sources: src/meltano/schemas/meltano.schema.json:344-380, integration/example-library/meltano-basics/index.md:23-25
Project Configuration and Management
Meltano projects are defined by a meltano.yml file, which follows a strict JSON schema. This file acts as the declarative source of truth for the project.
Sources: src/meltano/schemas/meltano.schema.json:1-10
Core Project Settings
Key settings defined at the project level include:
- requires_meltano: Version specifier to ensure compatibility.
- database_uri: The system database for storing job logs and state (defaults to SQLite).
- state_backend: Configuration for where state is stored (System DB, S3, GCS, or Azure).
- environments: Array of environment-specific overrides (e.g.,
dev,staging,prod).
Sources: src/meltano/schemas/meltano.schema.json:11-100
The Manifest System
The Meltano Manifest is a compiled, pre-processed JSON representation of a Meltano project. It resolves all include_paths and plugin lockfiles into a single source of truth. Manifests are environment-specific (e.g., meltano-manifest.prod.json).
The processing follows a strict hierarchy where environment variables are merged and expanded from the terminal level down to the specific plugin execution environment.
Sources: src/meltano/core/manifest/init.py:1-40
Technical Stack
Meltano is developed using modern Python standards and robust development tools.
- Language: Python 3.10+ (support up to 3.14).
- Package Management:
uvis used for high-performance dependency management and virtual environment creation. - Task Running:
noxmanages development tasks like linting and testing. - Linting/Formatting:
ruffis the primary tool for code style enforcement. - Testing:
pytestwith support for multiple database backends (SQLite, PostgreSQL).
Sources: AGENTS.md:7-13, CHANGELOG.md:144-150
Basic Workflow
The standard operational workflow in Meltano involves installing defined plugins, configuring them, and then running tasks.
- Installation:
meltano installcreates isolated virtual environments for every plugin. - Configuration:
meltano config <plugin> set <key> <value>manages plugin settings, supporting sensitive data redaction. - Execution:
meltano run <extractor> <loader>(ormeltano elt) executes the data movement.
Sources: integration/example-library/meltano-basics/index.md:7-25, CHANGELOG.md:195-200
Conclusion
Meltano provides a code-first framework for data engineering that prioritizes isolation (via virtual environments), portability (via declarative YAML/JSON), and reliability (via manifest-based state management). By standardizing the interface for hundreds of different data tools, it allows teams to manage complex ELT pipelines with software engineering best practices.
Sources: README.md:21-25, AGENTS.md:43-50