Installer Downloads and Validation

Relevant source files

The following files were used as context for generating this wiki page:

• builder/builder/download.py
• builder/builder/sources/adapters.py
• builder/builder/main.py
• builder/builder/opsi.py
• README.md
• tests/test_workflow.py

The installer download and validation system manages the secure retrieval of Windows application installers from upstream vendors. It ensures that every binary entering the private OPSI repository matches its published metadata, follows strict security policies, and maintains cryptographic integrity over time.

This module acts as the gatekeeper for the repo-builder component. It implements protocol-specific adapters to resolve the latest versions, performs multi-layered validation of the downloaded payload, and generates provenance data used for long-term state tracking. Sources: README.md:1-15, builder/builder/main.py:25-45

Metadata Resolution and Source Adapters

Before a download begins, the system resolves the specific version and URL for a package using metadata adapters. These adapters interpret various vendor APIs and formats to extract download links and expected SHA256 checksums.

Resolution Flow

The system supports multiple source types, including GitHub releases, direct URLs with regex matching, Mozilla's version API, Microsoft's update API, and WinGet manifests. Each adapter enforces a allowed_hosts policy to prevent unauthorized redirects.

The diagram shows how the system branches based on the source type defined in the catalog to produce a standardized Release object. Sources: builder/builder/sources/adapters.py:65-180, tests/test_workflow.py:75-100

Key Adapter Functions

Function	Description
resolve()	The main entry point that dispatches to specific adapters based on catalog configuration.
select_github()	Filters GitHub releases for stable assets matching a specific architecture (default x64).
validate_url()	Enforces HTTPS-only schemes and pins hostnames to the allowed_hosts list.
field()	Navigates nested JSON paths to extract version or checksum data.

Sources: builder/builder/sources/adapters.py:15-60, builder/builder/download.py:15-30

Secure Download Implementation

The download process uses a specialized HTTP class that provides robust error handling, rate limiting, and security checks.

Rate Limiting and Retries

The builder respects upstream rate limits, specifically for the GitHub API. If a 403 or 429 status code is received, the system extracts the X-RateLimit-Reset header and defers further attempts for that host until the limit resets. General network errors trigger up to three retries with an exponential backoff. Sources: builder/builder/download.py:33-55, builder/builder/download.py:145-155

Download Lifecycle

The sequence diagram illustrates the chunked streaming process which allows real-time hash calculation and early termination if size limits are exceeded. Sources: builder/builder/download.py:70-110, builder/builder/download.py:170-195

Validation and Integrity Guards

The system performs multiple checks to ensure the binary is valid and safe before it is packaged into an .opsi archive.

Verification Layers

1. Host Validation: Rejects any URL or redirect that does not match the allowlist or uses insecure protocols.
2. Size Limits: Aborts if the Content-Length or actual downloaded bytes exceed MAX_DOWNLOAD_BYTES (default 2 GiB).
3. Format Validation: Verifies file signatures (magic bytes). For example, .exe files must start with MZ, and .msi files must match the OLE compound file header.
4. Cryptographic Integrity: Compares the calculated SHA256 hash against the metadata resolved from the source adapter.
5. Digital Signatures: Uses osslsigncode to check if the binary is digitally signed, though it does not enforce specific publishers by default.

Sources: builder/builder/download.py:180-215, README.md:320-340

Immutable Checksum Protection

To prevent supply chain attacks or accidental version overwrites, the system maintains a "checksum history" in state/packages.json. If a version is already known, the builder refuses to download a binary with a different hash unless the ALLOW_CHANGED_CHECKSUM environment variable is explicitly set to true.

def protect_checksum(previous, actual, allow=False):
    if previous and previous != actual and not allow:
        raise ValueError(
            "IMMUTABLE VERSION CHECKSUM CHANGED; refusing rebuild (ALLOW_CHANGED_CHECKSUM=false)"
        )

Sources: builder/builder/download.py:126-131, builder/builder/main.py:100-115

Provenance and Metadata Generation

Upon successful validation, the system generates a provenance.json file. This file acts as a permanent record of the installer's origin and is bundled inside the OPSI package at CLIENT_DATA/provenance.json.

Provenance Data Structure

Field	Description
sha256	The SHA256 hash of the original installer.
source_url	The exact URL from which the installer was retrieved.
signature_status	One of SIGNED, UNSIGNED, or UNKNOWN.
download_timestamp	ISO timestamp of the retrieval.
package_sha256	The hash of the resulting .opsi archive.

Sources: builder/builder/main.py:130-150, builder/builder/opsi.py:80-95

Conclusion

The Installer Download and Validation system provides a high-security pipeline for binary acquisition. By combining strict host allowlisting, immutable checksum enforcement, and automated signature verification, the project ensures that the private OPSI repository remains a trusted source for internal software distribution. The use of provenance metadata allows administrators to audit the origin of any package and recover state even in the event of primary database loss. Sources: README.md:400-415, builder/builder/main.py:215-230