Python Project Configuration
Relevant source files
The following files were used as context for generating this wiki page:
The Python project configuration for the Meltano Extension Developer Kit (EDK) establishes the foundational environment for building, packaging, and templating Meltano extensions. It utilizes modern Python standards, specifically employing hatchling as the build backend and copier for project scaffolding. This configuration ensures consistent development environments, robust logging standards, and a predictable lifecycle for extension execution.
The configuration scope includes dependency management, build system definitions, and a template-driven approach to generating new extensions with pre-configured CLI prefixes, library names, and metadata.
Sources: pyproject.toml, copier.yml, README.md
Build System and Metadata
The project is configured as a PEP 621 compliant Python package. It uses the hatchling build backend with hatch-vcs for dynamic versioning based on version control system tags.
Core Metadata and Dependencies
The project metadata defines the package identity as meltano.edk, requiring Python 3.10 or newer. It lists several key dependencies required for extension functionality:
| Dependency | Version Requirement | Purpose |
|---|---|---|
structlog | >=21,<27 | Structured logging framework |
PyYAML | >=6,<7 | YAML parsing and serialization |
devtools | >=0.9.0,<1 | Pretty printing for development |
Sources: pyproject.toml:1-55
Build and Linting Tools
The configuration utilizes ruff for linting and formatting, with a specific set of rules including E (Error), F (Pyflakes), B (flake8-bugbear), and I (isort). The build targets are defined for both sdist and wheel formats, specifically including the meltano package directory.
Sources: pyproject.toml:90-136
Extension Scaffolding with Copier
The Meltano EDK includes a copier template to standardize the creation of new extensions. This configuration allows developers to initialize a project with pre-defined structures and variables.
Template Variables
The copier.yml file defines the input variables used to customize the generated project configuration:
| Variable | Type | Description |
|---|---|---|
extension_name | string | Proper name of the extension (e.g., MyExtension) |
admin_name | string | Author name for pyproject.toml |
cli_prefix | string | Prefix used for the generated CLI tool |
wrapper_target_name | string | The command being wrapped (e.g., dbt) |
Sources: copier.yml:1-50
The following diagram illustrates the project initialization flow using the configuration:
The diagram shows how variables from copier.yml are processed to generate project-specific configuration files.
Sources: copier.yml:1-50, README.md:20-33
Runtime Configuration and Logging
The EDK provides standardized logging configurations that extensions can adopt to remain consistent with the Meltano ecosystem. This is managed through the meltano.edk.logging module.
Logging Environment Variables
Extensions can be configured at runtime using specific environment variables to control logging behavior:
| Environment Variable | Default | Description |
|---|---|---|
LOG_LEVEL | "INFO" | Sets the logging threshold (DEBUG, INFO, etc.) |
LOG_TIMESTAMPS | "False" | Includes ISO timestamps in logs if True |
LOG_LEVELS | "False" | Includes the log level name in logs if True |
MELTANO_LOG_JSON | "False" | Switches between Console and JSON rendering |
Sources: meltano/edk/logging.py:126-140
Logging Initialization Flow
The configuration process for structured logging involves a sequence of processor attachments:
The sequence diagram details the initialization of the logging system based on environmental configuration.
Sources: meltano/edk/logging.py:75-124
Command and Interface Models
The EDK configuration defines how extension commands are structured through Python dataclasses. This provides a formal model for the extension's capabilities, which can be exported as JSON or YAML via the describe method.
Extension Command Model
The ExtensionCommand dataclass defaults to a standard set of supported lifecycle commands:
describeinvokepre_invokepost_invokeinitialize
Sources: meltano/edk/models.py:22-35, meltano/edk/extension.py:102-125
Output Formats
The project configuration supports multiple output formats for extension metadata, controlled by the DescribeFormat enum:
text: Human-readable viadevtools.prettierjson: Standard JSON viajson.dumpsyaml: YAML viayaml.dump
Sources: meltano/edk/extension.py:22-28, meltano/edk/extension.py:102-125
Conclusion
The Python project configuration of the Meltano EDK provides a structured framework for extension development. By integrating hatchling for builds, copier for scaffolding, and structlog for runtime observability, it ensures that all extensions built with the EDK follow a predictable and maintainable architectural pattern. This configuration-heavy approach minimizes boilerplate for developers while maintaining compatibility with the broader Meltano platform.