Repository Metadata Management

Relevant source files

The following files were used as context for generating this wiki page:

• builder/builder/repository.py
• builder/builder/main.py
• builder/builder/opsi.py
• README.md
• scripts/audit-repository.py
• builder/builder/state.py

Repository Metadata Management maintains the integrity and availability of the OPSI 4.3 software repository. This system automates the generation of official OPSI metadata files, handles atomic publication of new packages, and enforces version retention policies to prevent disk exhaustion. The system ensures that OPSI clients and servers receive consistent and verified information about available .opsi archives through structured metadata files like packages.json and packages.msgpack.zstd.

Sources: README.md:1-20, builder/builder/repository.py:1-5

Metadata Generation Architecture

The repository builder generates metadata outside the live served directory to prevent in-flight readers from observing corrupted or partial data. It utilizes a temporary staging area within the work/ directory to construct the metadata files using official OPSI CLI tools.

Staging and Atomic Replacement

The system performs the following steps during metadata updates:

1. It creates a temporary directory in work/ prefixed with metadata-.
2. It links or copies all valid .opsi packages (excluding those marked for deletion) into this staging directory.
3. It executes opsi-cli manage-repo metafile create and scan-packages against the staging area.
4. It moves the resulting metadata files into the repository using atomic renames via temporary hidden files (e.g., .packages.json.tmp).

Sources: builder/builder/repository.py:32-52, README.md:20-25

The diagram shows the isolated flow of metadata generation to ensure repository consistency.
Sources: builder/builder/repository.py:32-52

Core Components and Files

The metadata management system tracks several file types within the repository directory to support OPSI clients and auditing scripts.

File Suffix	Purpose	Source
.opsi	The primary OPSI software package archive.	repository.py:9
.opsi.md5	MD5 checksum for OPSI transfer compatibility.	repository.py:59
.opsi.zsync	Zsync control file for efficient partial downloads.	repository.py:59
.opsi.provenance.json	Immutable evidence of source, checksum, and build timestamp.	main.py:100-117
packages.json	Repository-wide index of available products and versions.	scripts/audit-repository.py:8
packages.msgpack.zstd	Compressed binary repository index.	README.md:20

Sources: builder/builder/repository.py:58-62, scripts/audit-repository.py:7-20, README.md:20-25

Package Retention and Cleanup

The retain function enforces version limits to manage disk space while keeping a configurable number of previous versions available for recovery. By default, KEEP_VERSIONS is set to 2.

Deletion Logic

1. The obsolete function identifies packages exceeding the KEEP_VERSIONS threshold for each product ID.
2. Metadata is regenerated for the repository before any files are physically deleted.
3. Once metadata is safely published, the obsolete .opsi archives and their four associated sidecar files (.md5, .zsync, .provenance.json) are unlinked.

Sources: builder/builder/repository.py:24-30, builder/builder/repository.py:55-62, builder/builder/main.py:37

Integrity Auditing

The scripts/audit-repository.py tool provides a read-only validation of the repository state. It compares published packages against their provenance and metadata records to detect corruption.

Audit Checklist

• SHA256 Validation: Matches the current .opsi file hash against the package_sha256 field in its .provenance.json.
• MD5 Integrity: Verifies the .opsi file against the official OPSI .md5 sidecar.
• Zsync Check: Ensures .zsync files are present and non-empty.
• Metadata Consistency: Confirms that every entry in the repository's packages.json points to a file that exists and matches the recorded size.

Sources: scripts/audit-repository.py:10-22

Publication and Provenance

When a new package is successfully built, the publish function moves the artifact into the live repository. This operation is performed per file rather than for the entire repository at once.

def publish(package, repository):
    for suffix in (".md5", ".zsync", ".provenance.json", ""):
        source = Path(str(package) + suffix)
        target = Path(repository) / (package.name + suffix)
        hidden = target.with_name("." + target.name + ".tmp")
        shutil.copyfile(source, hidden)
        with hidden.open("rb") as f:
            os.fsync(f.fileno())
        os.replace(hidden, target)

Sources: builder/builder/repository.py:65-73

This procedure ensures that files are fully written to disk and synced before they replace existing versions, minimizing the risk of partial file reads by OPSI clients. The provenance.json file is bundled inside the archive's CLIENT_DATA and also placed alongside the archive in the repository to provide an immutable record of the installer's origin and signature status.

Sources: builder/builder/main.py:100-117, builder/builder/repository.py:65-73

Summary

Repository Metadata Management ensures the stability of the automated OPSI repository by isolating metadata generation, using atomic filesystem operations for publication, and maintaining strict version retention. By coupling the official OPSI metadata format with custom provenance records, the system provides both OPSI compatibility and enhanced auditability for software deployments.