Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Numan Registry testing strategy focuses on maintaining the integrity of a data-only repository through automated CI gates, static analysis, and multi-stage validation scripts. Because the project serves as a trust root for the Numan package manager, testing emphasizes security invariants, schema compliance, and artifact hash verification.
Testing is divided into repository safety checks (secret scanning and workflow pinning), unit testing of the Python toolchain, and a rigorous multi-stage intake process that ensures packages are compatible with the Nushell environment before they are promoted to the production index.
CI and Repository Safety
CI gates are the primary defense against registry corruption and security breaches. Every Pull Request triggers a suite of checks that validate the repository's state without requiring access to production secrets.
Security and Workflow Invariants
Static safety checks ensure that GitHub Actions workflows are secure and that no sensitive information is leaked.
- Action Pinning: All actions must be pinned to a 40-character commit SHA to prevent supply chain attacks.
- Permissions: Workflows default to
contents: readto maintain a principle of least privilege. - Secret Isolation: Production secrets (e.g.,
NUMAN_REGISTRY_PRIVATE_KEY) are restricted to protected environments and specific jobs. - Secret Scanning: The
scan_for_secrets.pyscript is used to prevent the accidental commitment of private keys or PEM files.
Sources: scripts/test_workflow_safety.py:27-64, AGENTS.md:20-25, REVIEW.md:38-40
The diagram shows the sequence of repository safety checks performed during CI to ensure environment integrity.
Multi-Stage Intake Validation
The testing strategy relies heavily on scripts/validate_candidate.py, which orchestrates several sub-tools to verify package specifications before they enter the registry.
Intake Validation Steps
| Step | Tool | Description |
|---|---|---|
| 1. Download & Hash | add-package.py | Downloads artifacts and computes SHA256 digests. |
| 2. Structural Lint | lint_packages.py | Performs structural checks on the candidate entry. |
| 3. Schema Check | validate.py | Verifies the JSON against schemas/index-v1.json. |
| 4. Lifecycle Prove | lifecycle-prove.py | (Optional/Strict) Installs and runs the package against a real Nu binary. |
Sources: scripts/validate_candidate.py:16-25, README.md:104-125
Lifecycle Proving
For "activatable" packages (plugins and modules with activation sections), a successful real-world execution is mandatory before production promotion. This test must be run on a clean root with a Nushell version matching the package's constraint.
Sources: scripts/test_validate_candidate.py:238-278, README.md:120-125
The sequence diagram illustrates the artifact verification and lifecycle execution flow for new package candidates.
Toolchain Unit Testing
The Python scripts that manage the registry are covered by a suite of unit tests using the unittest framework. These tests simulate various edge cases, such as malformed discovery reports or invalid semver constraints, to ensure the reliability of the automation.
Key Test Suites
test_gen_candidate.py: Validates the heuristics used to match filenames to target triples (e.g., mappingwin-x64tox86_64-pc-windows-msvc). Sources: scripts/test_gen_candidate.py:46-68test_sync_intake_candidates.py: Verifies the classification of artifact provenance (mirror vs. upstream vs. ci-built). Sources: scripts/test_sync_intake_candidates.py:25-50test_open_intake_pr.py: Ensures that PR generation logic correctly formats the intake summary and handles Git branch operations safely. Sources: scripts/test_open_intake_pr.py:44-70
Sources: REVIEW.md:16, CLAUDE.md:106-110
Parser Parity Testing
A critical component of the testing strategy is ensuring that the index.json remains compatible with the production Rust parser used by the Numan client.
CI runs cargo run --locked --manifest-path tools/numan-parser-check/Cargo.toml -- registry/index.json to verify that any changes to the catalog do not break the production parser. This prevents "silent index corruption" where the JSON is schema-valid but unparseable by the compiled client.
Sources: AGENTS.md:28-30, REVIEW.md:52-53
Summary
The Numan Registry testing strategy is a defense-in-depth approach tailored for a trust-root data repository. It combines static safety checks for infrastructure, unit tests for management tooling, and a strict, evidence-based intake process for package data. By requiring lifecycle evidence and Rust-parser parity, the project ensures that every update to the official registry is both secure and functional for end-users.