Manifests & Compilation
Relevant source files
The following files were used as context for generating this wiki page:A Meltano manifest is a JSON file that provides a complete, unambiguous, and static representation of a Meltano project. It serves as a pre-processed version of the meltano.yml project file, where complex logic—such as merging included files, resolving environment-specific overrides, and integrating plugin lockfiles—is performed during a compilation step rather than at runtime. This allows Meltano to use the manifest as a single source of truth that is self-contained and ready for execution in specific environments.
The compilation process is primarily handled by the meltano compile command. It transforms the declarative YAML configuration into a structured JSON format that adheres to the Meltano project schema. Manifests are designed to be portable; they sanitize machine-specific absolute paths by replacing them with the ${MELTANO_PROJECT_ROOT} variable, ensuring that the compiled configuration can function across different host environments.
Sources: src/meltano/core/manifest/init.py:3-23, src/meltano/core/manifest/manifest.py:165-176
Manifest Architecture and Structure
Core Characteristics
Manifest files are always environment-specific. Their naming convention follows the pattern meltano-manifest.<environment>.json. If no environment is active, the file is simply named meltano-manifest.json. Unlike standard project files, manifests do not contain an environments field because the environment-specific configuration has already been merged into the top-level project settings during compilation.
Sources: src/meltano/core/manifest/init.py:12-16, tests/meltano/cli/test_compile.py:64-70
Compilation Pipeline
The compilation involves a multi-stage merger of data from various sources within the Meltano project. The Manifest class in meltano.core.manifest.manifest orchestrates this logic.
The diagram above illustrates the transformation of raw project files and metadata into a final, validated manifest.
Sources: src/meltano/core/manifest/manifest.py:165-316
Data Merging and Precedence
During compilation, Meltano follows a strict hierarchy for merging environment variables and plugin configurations. The environment-level plugin configuration is considered the highest precedence.
| Level | Source | Description |
|---|---|---|
| 1 (Highest) | Environment Plugin Env | Variables defined inside a specific plugin within an environment. |
| 2 | Root Plugin Env | Variables defined for a plugin at the root level of meltano.yml. |
| 3 | Environment Env | Variables defined at the environment level. |
| 4 | Root Env | Variables defined at the project root level. |
Sources: src/meltano/core/manifest/manifest.py:270-282
The Compilation Process
CLI Implementation
The compilation is triggered via the meltano compile command. By default, Meltano attempts to compile a manifest for every environment defined in the project, as well as one for the "no environment" state.
Sources: src/meltano/cli/compile.py:44-53
Command Options
The compilation command supports several flags to control the output and validation:
| Flag | Type | Default | Description |
|---|---|---|---|
--directory | Path | .meltano/manifests | The output directory for compiled JSON files. |
--indent | Integer | 4 | The indentation level for the output JSON. |
--unsafe | Boolean | False | If True, prevents the redaction of secret values. |
--lint | Boolean | False | Enables schema validation for the compiled manifest. |
Sources: src/meltano/cli/compile.py:17-42, integration/example-library/meltano-manifest/index.md:12-25
Path Sanitization
To ensure portability, the manifest replaces absolute paths matching the project root with the ${MELTANO_PROJECT_ROOT} placeholder. This is handled by the sanitize_env_vars method.
Sources: src/meltano/core/manifest/manifest.py:230-244
Implementation Details
The Manifest Class
The Manifest class is the central component responsible for generating the manifest data. It uses a YamlNoTimestampSafeLoader to ensure that YAML timestamps remain as raw strings, preventing issues with JSON serialization.
class Manifest:
def __init__(
self,
project: Project,
path: Path,
*,
check_schema: bool,
redact_secrets: bool = False,
) -> None:
self.project = project
self._project_root_str = str(self.project.root.resolve())
self._meltano_file = self.project.meltanofile.read_text()
self.path = path
self.check_schema = check_schema
self.redact_secrets = redact_secrets
# ... schema loading ...
Sources: src/meltano/core/manifest/manifest.py:112-145
Schema Validation
If the --lint flag is used (or check_schema is True), the manifest is validated against the official meltano.schema.json. Validation errors are caught and logged as warnings, showing the exact JSON path of the violation (e.g., ::$.plugins.extractors.0.name).
Sources: src/meltano/core/manifest/manifest.py:155-181, tests/meltano/cli/test_compile.py:133-146
Scaffolding
Meltano uses "scaffolding" to ensure the manifest contains all expected keys (like plugins, extractors, loaders, etc.) even if they are empty in the source files. This simplifies consumption of the manifest by downstream tools. The apply_scaffold function recursively builds this structure based on the schema's requirements.
Sources: src/meltano/core/manifest/manifest.py:353-388
Usage Examples
Compiling for a Specific Environment
To compile a manifest only for the production environment:
meltano --environment=prod compile
Sources: integration/example-library/meltano-manifest/index.md:10-14
Compiling with Unsafe Secrets
By default, sensitive settings are replaced with (redacted). To include actual secret values (e.g., for local debugging or CI/CD pipelines where the file is protected):
meltano compile --unsafe
Sources: src/meltano/cli/compile.py:34-37, tests/meltano/cli/test_compile.py:192-205
Conclusion
Manifests and compilation provide a robust way to bridge declarative configuration and executable state. By resolving dependencies and environment variables at compile-time, Meltano ensures that execution is predictable, portable, and decoupled from the complexities of the original YAML file structure. Manifests serve as the foundation for the future simplification of Meltano Core, aiming to reduce the overhead of runtime configuration management.