Publishing to PyPI
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) uses the hatchling build backend to manage the packaging and distribution of extensions to the Python Package Index (PyPI). This process ensures that Meltano extensions are correctly versioned, bundled with necessary metadata, and compatible with the broader Python ecosystem.
Build System Configuration
The project's build infrastructure is defined in pyproject.toml. It specifies hatchling as the build backend and utilizes hatch-vcs for dynamic versioning based on Version Control System (VCS) tags.
Build Backend Components
| Component | Description |
|---|---|
hatchling | The primary build backend used to create source distributions and wheels. |
hatch-vcs | A plugin that allows the package version to be determined dynamically from Git tags. |
sdist | Target for source distributions, specifically including the meltano package directory. |
wheel | Target for binary distributions, ensuring the meltano package is bundled. |
Sources: pyproject.toml:105-117, pyproject.toml:92-98
Metadata and Packaging Requirements
For an EDK-based project to be successfully published, it must define specific metadata in its pyproject.toml. This metadata categorizes the project on PyPI and defines its requirements.
Essential Metadata Fields
- Project Name: Defined as
meltano.edkfor the core library. - Dynamic Versioning: The version is not hardcoded but managed via VCS.
- Dependencies: Core dependencies required for functionality include
structlog,PyYAML, anddevtools. - Classifiers: The project includes Trove classifiers indicating Python compatibility (3.10 through 3.14) and license type (Apache-2.0).
Sources: pyproject.toml:3-33
Dependency Groups
The EDK distinguishes between runtime dependencies and development-time dependencies. While dependencies are included in the PyPI distribution, dependency-groups like dev (containing copier, pytest, ruff, and mypy) are used only for local development and testing.
Sources: pyproject.toml:68-76
Packaging Flow
The following diagram illustrates the flow from source code and metadata to a distributable package ready for PyPI.
The build process consumes the local source code and VCS information to produce standardized distribution formats.
Sources: pyproject.toml:92-117
Copier Template for New Extensions
The EDK provides a copier template that pre-configures the packaging structure for new extensions. This ensures that any extension built with the EDK is immediately ready for distribution with a standardized pyproject.toml.
Template Configuration
The copier.yml file defines prompts that populate the packaging metadata:
extension_name: Used to generate the package name.admin_name: Populates the author field in the generatedpyproject.toml.library_name: Defines the internal Python package name.
Sources: copier.yml:1-30
Validation of Generated Packages
The EDK includes tests to ensure that the code generated by the template meets quality standards (using ruff and mypy) before a developer attempts to package or publish it.
# Use ruff to check the generated code
ruff_check = [
"ruff",
"check",
outdir,
"--select",
"E,F,W,Q",
]
p = subprocess.run(ruff_check, capture_output=True)
assert p.returncode == 0, p.stdout.decode("utf-8")
Sources: tests/copier_template/test_generated.py:53-62
Summary
Publishing Meltano EDK extensions to PyPI relies on the hatchling build system and hatch-vcs for version management. By utilizing the provided copier template, developers can generate projects that adhere to these packaging standards, ensuring compatibility and ease of distribution.