Workflow Safety Testing
Relevant source files
The following files were used as context for generating this wiki page:
Workflow Safety Testing in the numan-plugins project ensures the integrity, security, and immutability of the CI build and release pipeline. The primary purpose is to maintain a strict trust boundary where the repository builds and hosts binaries without possessing signing keys, while preventing unauthorized or accidental publication of assets.
The scope of these tests covers workflow dispatch triggers, GitHub Action pinning, permission management, and the validation of "Draft-then-finalize" release transactions. These measures ensure that release assets are immutable and that the publication process remains manual and strictly controlled.
Sources: README.md:21-31, REVIEW.md:20-35, scripts/test_workflow_safety.py:46-55
Core Safety Invariants
The safety architecture is built around several non-negotiable invariants designed to protect the registry and users from malicious or accidental changes.
| Invariant | Description | Source |
|---|---|---|
| Manual Dispatch Only | Publication must only occur via manual build-plugins dispatch with a non-empty package list. | REVIEW.md:33, scripts/test_workflow_safety.py:46-55 |
| Action Pinning | Every GitHub Action must be pinned to a 40-character commit SHA to prevent supply-chain attacks. | scripts/test_workflow_safety.py:57-63 |
| Immutable Assets | Existing release tags and assets cannot be overwritten; changes require new versions. | README.md:26-28, REVIEW.md:25-26 |
| Draft-then-Finalize | Releases are assembled as run-owned drafts and only made public after full verification. | README.md:28-30, REVIEW.md:27-28 |
| No Private Keys | Signing keys and registry secrets are never stored in this repository. | REVIEW.md:22-23, README.md:21-22 |
Release Transaction Logic
The project employs a transaction-based approach to releases to ensure that no "half-published" or "orphaned" releases exist. This logic is implemented in scripts/release_transaction.py and verified by scripts/test_workflow_safety.py.
Transaction Phases
The transaction consists of three primary phases: Claim, Upload, and Finalize. If any step fails before finalization, a Cleanup operation is triggered to remove the draft and associated tag.
This diagram illustrates the sequential stages of a release transaction and the fail-safe cleanup path.
Sources: scripts/release_transaction.py:165-201, scripts/test_workflow_safety.py:82-89
Asset Verification
Before a release is finalized, the system performs a strict comparison between local assets and the assets present in the GitHub draft. This includes checking file names, sizes, and SHA-256 digests.
def finalize(repo, release_id, tag, commit, assets_dir, runner):
# ... validation of release draft status and tag commit ...
actual = {
asset["name"]: {"size": asset["size"], "digest": asset.get("digest")}
for asset in release.get("assets", [])
}
expected = expected_assets(assets_dir)
if actual != expected:
raise ValueError(f"release asset set mismatch: expected {expected}, got {actual}")
Sources: scripts/release_transaction.py:129-155
Workflow Integrity Checks
Static and dynamic checks are performed on the .yml workflow files to prevent common security pitfalls.
Shell Expression Interpolation
To prevent script injection, the test_publication_shell_never_interpolates_expressions test ensures that GitHub Actions expressions (e.g., ${{ }}) are never used directly within executable shell text (run: blocks). Instead, they must be passed via environment variables or with fields.
Sources: scripts/test_workflow_safety.py:65-80
Permission Scoping
Permissions are strictly audited. Only the specific release job is permitted to have contents: write access, which is required for creating releases and uploading assets. All other jobs and global permissions are restricted to contents: read.
Sources: scripts/test_workflow_safety.py:82-89
Matrix and Environment Safety
Steps that expand $MATRIX_* environment variables are forced to use the bash shell. This prevents issues on Windows runners where pwsh might incorrectly handle or empty these variables during expansion.
Sources: scripts/test_workflow_safety.py:101-120
Pre-Publication Validations
Before any build starts, several validation scripts verify the state of the repository and the requested changes.
Manifest and Tag Verification
The scripts/validate_manifest.py script ensures that:
- Every active plugin entry contains all required fields (repo, name, owner, source_commit, etc.).
- The
source_commitis a valid 40-character hex string. - If
--verify-upstreamis used, the human-facingtagin the manifest actually resolves to the declaredsource_commitin the upstream repository.
The sequence diagram shows how the manifest validator interacts with upstream repositories to ensure provenance.
Sources: scripts/validate_manifest.py:114-142, scripts/validate_manifest.py:145-161
Absolute Absence Check
The scripts/ensure_release_absent.py script is executed to "fail closed" if a tag or release already exists for the target version. This enforces asset immutability.
Sources: scripts/ensure_release_absent.py:46-64
Summary of Safety Components
| Component | Responsibility | Relevant Files |
|---|---|---|
| Manifest Validator | Checks schema and upstream tag-to-commit mapping. | scripts/validate_manifest.py |
| Release Transaction | Handles atomic draft creation, upload, and finalization. | scripts/release_transaction.py |
| Workflow Linter | Enforces SHA pinning and trigger restrictions. | scripts/test_workflow_safety.py |
| Absence Guard | Prevents overwriting existing releases. | scripts/ensure_release_absent.py |
Conclusion
Workflow Safety Testing in numan-plugins serves as the primary defense against supply-chain compromises and delivery errors. By combining strict manifest validation, atomic release transactions, and rigorous workflow auditing, the project ensures that every binary delivered to the numan registry is traceable to a specific, verified upstream commit and has not been tampered with during the build process.