Testing Strategy & Overview

Relevant source files

The following files were used as context for generating this wiki page:

The testing strategy for numan-plugins is designed to ensure the integrity, safety, and reproducibility of the plugin build and delivery pipeline. Because this repository serves as a feeder for the official numan-registry, testing focuses heavily on verifying the trust boundary, ensuring immutable release assets, and validating the automation scripts that handle packaging and release transactions.

The strategy encompasses unit testing of Python scripts, CI/CD workflow safety checks, and manual lifecycle verification. The primary goal is to ensure that source-only Nushell plugins are correctly cross-compiled, packaged into normalized archives, and published without compromising the security of the registry.
Sources: README.md:1-25, REVIEW.md:7-15

Core Testing Components

The project utilizes a multi-layered testing approach to validate different aspects of the pipeline.

Unit Testing and Script Validation

All primary automation scripts located in the scripts/ directory have corresponding test files. These tests use the standard Python unittest framework to verify functional correctness and error handling.

Test File	Target Module	Primary Responsibility
`test_package_plugin.py`	`package_plugin.py`	Verifies deterministic archive creation (tar.gz/zip).
`test_validate_manifest.py`	`validate_manifest.py`	Validates schema and upstream tag-to-commit resolution.
`test_release_transaction.py`	`release_transaction.py`	Mocks GitHub CLI to test claim/finalize/cleanup logic.
`test_gen_spec_source.py`	`gen_spec.py`	Ensures metadata and provenance are correctly emitted in specs.
`test_ensure_release_absent.py`	`ensure_release_absent.py`	Confirms scripts fail closed if a tag/release already exists.

Sources: scripts/test_package_plugin.py, scripts/test_validate_manifest.py, scripts/test_release_transaction.py

Workflow and Repository Safety

A dedicated test suite, test_workflow_safety.py, parses the repository's GitHub Action YAML files to enforce security invariants.

Manual Dispatch Only: Ensures that publication can only be triggered by workflow_dispatch with an explicit only= filter, preventing accidental releases from Pushes or PRs.
Action Pinning: Verifies that all uses: statements in workflows are pinned to a 40-character commit SHA rather than mutable tags.
Permission Isolation: Validates that only the specific release job has contents: write permissions.
Shell Safety: Scans shell scripts within workflows to ensure they do not interpolate GitHub expressions directly into executable text, mitigating injection risks.

Sources: scripts/test_workflow_safety.py:44-90, REVIEW.md:17-25

Release Lifecycle Verification

The testing strategy includes a strict "Flow" for every plugin added to the registry to ensure target compatibility and registry integrity.

Generating architecture diagram...

The diagram illustrates the sequential verification steps from manifest entry to final garbage collection on a target OS.

Lifecycle Steps

Tag Verification: validate_manifest.py --verify-upstream resolves tags to commit SHAs via git ls-remote.
Deterministic Packaging: Archives are built with fixed mtime (1980-01-01) to ensure stable SHA256 hashes.
Transaction Safety: release_transaction.py claims a tag, creates a draft, uploads assets, and only finalizes if the remote asset set matches the local build exactly.
Target Verification: Plugins must pass a full lifecycle test (search, info, install, activate, doctor, list, remove, gc) against a real Nushell binary on all target Operating Systems.

Sources: README.md:58-71, scripts/release_transaction.py:125-150, scripts/package_plugin.py:27-45

Architectural Invariants for Testing

Reviewers and automated tests must flag violations of these architectural rules:

No Private Keys: The repository must never contain signing keys or PEM material.
Immutable Assets: Existing release tags or assets must never be overwritten; changes require a new version or build revision.
Provenance Retention: Generated specs must preserve source.rev as the immutable upstream commit to ensure binary provenance.
Hash Handling: Hashes are NOT hand-typed; they are generated during build for sanity checks and re-computed by numan-registry during intake.

Sources: REVIEW.md:27-45, scripts/gen_spec.py:118-135

Summary

The numan-plugins testing strategy prioritizes provenance and safety over simple functional coverage. By combining static analysis of GitHub workflows, deterministic build verification, and a strict release transaction protocol, the system ensures that third-party Nushell plugins are safely transformed into hash-pinned binaries suitable for the official registry. Success is defined by the ability to reconstruct identical artifacts from pinned upstream commits while maintaining a clear trust boundary that excludes signing keys from the build environment.

Sources: README.md:27-36, REVIEW.md:47-55