Python Script Unit Tests
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Python Script Unit Tests provide a rigorous verification layer for the automation and safety scripts used in the numan-plugins repository. These tests ensure that the build manifest is valid, packaging is deterministic, release transactions are atomic and reversible, and GitHub Actions workflows maintain strict security boundaries. Sources: README.md, REVIEW.md
The test suite is built primarily using the standard Python unittest framework and is executed as a mandatory CI gate. It covers critical path logic for plugin publication, including manifest verification, asset hash validation, and the "draft-then-finalize" release workflow. Sources: REVIEW.md:7-10, scripts/test_workflow_safety.py:118-124
Core Test Modules
Release Transaction Validation
The ReleaseTransactionTests verify the complex logic involved in claiming, uploading, and finalizing GitHub releases. These tests use a FakeRunner to simulate GitHub CLI (gh) responses, ensuring that failures during any stage of a release transaction result in a proper rollback. Sources: scripts/test_release_transaction.py:41-65
Key behaviors tested include:
- Atomic Claims: Ensuring a Git tag is created before a draft release is attempted. Sources: scripts/test_release_transaction.py:68-72
- Rollback Integrity: Verifying that if a release claim fails, the partially created tag is deleted. Sources: scripts/test_release_transaction.py:74-78
- Asset Verification: Enforcing that remote assets exactly match local builds in size and SHA-256 digest before publication. Sources: scripts/test_release_transaction.py:118-142
Manifest and Specification Integrity
Manifest tests ensure the manifest.json file adheres to structural requirements, such as valid 40-character lowercase hex source commits and unique plugin names. Sources: scripts/test_validate_manifest.py:77-85
The BuildSpecSourceTests verify that the generated registry specifications preserve source provenance, including the original repository URL and immutable commit revision. Sources: scripts/test_gen_spec_source.py:33-66
The following flowchart illustrates the manifest validation and spec generation testing flow:
Workflow and Repo Safety
The WorkflowSafetyTests enforce architectural invariants within the GitHub Actions YAML files. These tests ensure that the repository remains secure and that production builds are strictly controlled. Sources: scripts/test_workflow_safety.py:53-127
| Test Name | Security/Functional Requirement |
|---|---|
test_publication_is_manual_dispatch_only | Publication must only trigger via workflow_dispatch, never on PR or push. |
test_every_action_is_pinned_to_a_commit | Every GitHub Action must be pinned to a 40-character commit SHA, not a mutable tag. |
test_publication_shell_never_interpolates_expressions | Workflow expressions ${{ }} must never appear in executable shell text to prevent injection. |
test_only_release_job_can_write_contents | Only the specific release job has contents: write permissions. |
test_macos_uses_supported_runners | macOS builds must use macos-15 or macos-15-intel runners. |
Sources: scripts/test_workflow_safety.py:54-127, REVIEW.md:20-30
Packaging and Determinism
The PackagePluginTests ensure that the packaging scripts (package_plugin.py) produce byte-for-byte identical archives when given the same input. This is critical for maintaining stable SHA-256 hashes in the registry. Sources: scripts/test_package_plugin.py:30-51
Sources: scripts/test_package_plugin.py:33-41
Quality and Documentation Gates
The project enforces high code quality through specific structural tests:
- Docstring Coverage: The
DocstringCoverageTestsuse Python'sastmodule to verify that every module, function, and class in safety-critical scripts has an associated docstring. Sources: scripts/test_docstring_coverage.py:21-34 - Upstream Verification: Tests verify that scripts correctly resolve remote Git tags to commit SHAs to ensure the manifest pins are accurate. Sources: scripts/test_validate_manifest.py:97-101
Conclusion
Python Script Unit Tests serve as the primary defense mechanism for the numan-plugins delivery pipeline. By validating every stage from manifest authorship to release finalization, the suite ensures that the binaries hosted in this repository are secure, traceable, and reliably hash-pinned for the numan registry. Sources: README.md, REVIEW.md:46-51