Project Configuration (meltano.yml)

Relevant source files

The following files were used as context for generating this wiki page:

• src/meltano/schemas/meltano.schema.json
• src/meltano/core/plugin/file.py
• src/meltano/core/plugin/meltano_file.py
• src/meltano/core/manifest/init.py
• src/meltano/core/bundle/settings.yml
• src/meltano/core/bundle/initialize.yml

The meltano.yml file is the central declarative configuration file for a Meltano project. It defines the project's identity, metadata database connection, plugin definitions (extractors, loaders, etc.), pipeline schedules, and environment-specific overrides. This file serves as the source of truth from which a compiled manifest is generated for runtime execution.

Sources: src/meltano/schemas/meltano.schema.json:4-10, src/meltano/core/manifest/init.py:5-15

Project Settings

Global settings define the foundational behavior of the project, including version requirements, unique identifiers, and telemetry preferences.

Setting	Type	Default	Description
project_id	string	GUID	A globally unique identifier for the project.
requires_meltano	string	N/A	Version specifier (e.g., >=3,<4) for the required Meltano version.
database_uri	string	sqlite:///.../meltano.db	URI for the project system database.
send_anonymous_usage_stats	boolean	true	Whether to share anonymous data with the Meltano team.
default_environment	string	null	The environment to use if none is explicitly specified.

Sources: src/meltano/schemas/meltano.schema.json:28-66, src/meltano/core/bundle/settings.yml:4-36

Architecture and Data Flow

The configuration is structured hierarchically. At runtime, the project configuration can be compiled into a manifest, which resolves all environment variables and inheritance rules.

The diagram shows the flow from the static project configuration file through environment resolution to the final manifest used for execution.
Sources: src/meltano/core/manifest/init.py:5-48

Plugin Configuration

Plugins are categorized into types such as extractors, loaders, mappers, and utilities. Each plugin entry specifies its source (pip_url), unique configuration, and capabilities.

Extractor Configuration

Extractors include specific properties for data selection and catalog management.

• Capabilities: Singer features supported by the tap, such as catalog, discover, and state.
• Select: An array of entity/attribute selection rules (e.g., *.*).
• Metadata: Object for overriding Singer stream and property metadata.

Sources: src/meltano/schemas/meltano.schema.json:468-540

File Plugins

File plugins manage files within the project directory. They use an _update setting to control which managed files are automatically updated during upgrades.

# From src/meltano/core/plugin/file.py
EXTRA_SETTINGS: t.ClassVar[list[SettingDefinition]] = [
    SettingDefinition(
        name="_update",
        kind=SettingKind.OBJECT,
        aliases=["update"],
        value={},
    ),
]

Sources: src/meltano/core/plugin/file.py:34-42

Environments

Environments allow users to define sets of configuration overrides for different stages of development (e.g., dev, staging, prod).

The relationship shows how a Project contains multiple Environments, each with its own configuration and environment variable mappings.
Sources: src/meltano/schemas/meltano.schema.json:1152-1234, src/meltano/core/bundle/initialize.yml:1-12

Manifest and Compilation

A Meltano manifest is a pre-processed JSON representation of the project. Unlike the standard configuration, it is self-contained and environment-specific.

• Self-Contained: It merges all include_paths and lockfiles.
• Hierarchical Expansion: Environment variables are merged and expanded in a strict order: Terminal/.env -> Job Env -> Schedule Env -> Project Env -> Plugin Env.
• Validation: Manifests conform to the same schema as meltano.yml.

Sources: src/meltano/core/manifest/init.py:5-60

State Backend

Meltano supports various backends for storing pipeline state, including the system database and cloud storage (S3, GCS, Azure).

• URI: Defines the backend type (e.g., systemdb).
• Locking: Configurable lock_timeout_seconds and lock_retry_seconds to manage concurrent access.
• Cloud Specifics: Authentication details for S3 (aws_access_key_id), GCS (application_credentials), or Azure (connection_string).

Sources: src/meltano/schemas/meltano.schema.json:76-140, src/meltano/core/bundle/settings.yml:61-75

Conclusion

The meltano.yml file provides a declarative framework for data integration. By separating global settings, plugin definitions, and environment-specific overrides, it enables consistent pipeline behavior across different deployment targets while maintaining a single, version-controllable source of truth.