Archive Handling & Mirroring
Relevant source files
The following files were used as context for generating this wiki page:
Archive handling and mirroring are critical components of the Numan registry system, ensuring that Nushell packages (modules, scripts, and completions) are available in stable, verifiable formats. While plugins are typically distributed as platform-specific binaries, other package types often rely on source archives. The mirroring system provides a mechanism to host byte-stable versions of these archives when upstream sources do not provide compliant, permanent release assets.
The registry enforces strict rules on archive formats to ensure compatibility with the Numan client's extraction logic. Mirroring is specifically employed for "worth adding" candidates that lack byte-stable upstream assets, such as GitHub's automatically generated (and potentially mutable) source zips.
Sources: README.md:25-28, scripts/sync-intake-candidates.py:231-240
Supported Archive Formats
The registry strictly limits the archive extensions it will accept during intake. This alignment ensures that the Numan client can reliably extract packages across different operating systems.
| Extension | Description |
|---|---|
.zip | Standard compressed archive |
.tar.gz / .tgz | Gzip-compressed tarball |
.tar.xz / .txz | XZ-compressed tarball |
.tar | Uncompressed tarball |
Sources: scripts/archive_formats.py:7-14
Deterministic Mirroring Process
The mirroring system uses a specialized tool, build-mirror-zip.py, to create "byte-stable" archives from Git repositories. This is necessary because standard compression tools often produce different binary outputs for the same input files due to variations in timestamps or metadata.
Determinism Invariants
To achieve identical SHA-256 hashes across different build environments, the mirroring script enforces the following:
- Fixed Timestamps: All file entries in the zip are set to a constant date (1980-01-01 00:00:00).
- Fixed File Modes: Permissions are normalized to
0o644. - Constant System ID: The "Create System" ID is forced to Unix (3) to avoid hash drift between Windows and Linux builders.
Sources: scripts/build-mirror-zip.py:25-31, scripts/build-mirror-zip.py:108-119
Mirroring Workflow
The workflow involves cloning a specific Git reference, staging selected paths, and packaging them into a deterministic zip file.
The diagram shows the sequence from an upstream Git repository to a verified, byte-stable mirror asset ready for the registry.
Sources: scripts/build-mirror-zip.py:125-154
Artifact Classification and Provenance
The registry classifies artifacts based on their URL patterns to track trust and maintenance requirements. This is managed during the intake synchronization process.
- Mirror: Hosted on the
numan-registryreleases page (e.g.,numan-registry/releases/download/mirror-...). - CI-Built: Built and signed by the official plugin pipeline (e.g.,
numan-plugins/releases/...). - Upstream: Point directly to the original developer's release assets.
- Mixed: Packages where different versions or targets utilize different provenance sources.
Sources: scripts/sync-intake-candidates.py:33-54
Archive Metadata in the Index
Every archived package in the registry index must define specific metadata to allow the Numan client to locate and activate the contents.
Data Model for Archives
Archived artifacts utilize the following schema properties within registry/index.json:
| Field | Type | Description |
|---|---|---|
kind | Enum | Must be "archive". |
url | URI | The location of the archive. |
sha256 | String | Hex-encoded SHA-256 hash of the archive. |
entry | String | Path to the main .nu file relative to the archive root. |
archive_root | String | The top-level directory name inside the archive. |
Sources: schemas/index-v1.json:88-115
Implementation Example
A typical entry for a mirrored module demonstrates how the archive_root and entry paths work together to facilitate activation.
{
"version": "1.1.0",
"artifact": {
"kind": "archive",
"url": "https://github.com/tonythethompson/numan-registry/releases/download/mirror-vyadh-nutest-v1.1.0/nutest-1.1.0.zip%22%2C
"sha256": "f33b4892509a7328bd3d8648ffe1cd5b4b541f2492783c9d9f3f81b5fd7c2c79",
"entry": "nutest/mod.nu",
"archive_root": "nutest-1.1.0"
}
}
Sources: registry/index.json:62-68
Intake and Verification Rules
The intake process for archives follows a "Stage 4" generation logic where candidates are automatically derived from discovery reports.
- Selection: Prefer upstream release assets if they are byte-stable and in supported formats.
- Fallback: If no compliant upstream asset exists, flag for a registry-hosted mirror.
- Pathing: For modules, the system defaults the activation kind to
nu-moduleand sets the entry point tomod.nuby convention. - Security: Hashes must be computed from actual downloads via
scripts/add-package.pyand never hand-typed.
Sources: scripts/gen_candidate.py:116-125, scripts/sync-intake-candidates.py:231-236, README.md:43-44
Summary
Archive handling and mirroring provide the "source build" alternative for the Numan registry, focusing on Nushell-native code. By enforcing deterministic zip creation and strict format validation, the project ensures that even when upstream release practices vary, the registry provides a consistent and secure installation experience for all users. Mirroring is used as a bridge to convert mutable source snapshots into immutable, version-pinned assets.