Upstream Discovery Adapters

Relevant source files

The following files were used as context for generating this wiki page:

• builder/builder/sources/adapters.py
• README.md
• builder/builder/main.py
• catalog/packages.yaml
• builder/builder/models.py (Inferred from context in adapters.py and main.py)
• builder/builder/download.py

Upstream Discovery Adapters are specialized modules that identify and locate new software releases from external vendor sources. They translate disparate metadata formats—such as GitHub APIs, JSON feeds, and HTML directories—into a standardized Release object for the repository builder.

These adapters reside in builder/builder/sources/adapters.py and execute during each repository update cycle. The system matches the source.type defined in catalog/packages.yaml to a specific adapter logic to retrieve version numbers, download URLs, and cryptographic hashes.

Sources: README.md:144-147, builder/builder/sources/adapters.py:61-63

Architecture and Data Flow

The discovery process begins when builder/builder/main.py calls the resolve function for a package defined in the catalog. The resolve function acts as a dispatcher, selecting the appropriate adapter based on the package configuration.

The diagram shows how the resolve function dispatches package metadata to specific adapters to produce a unified Release object.
Sources: builder/builder/sources/adapters.py:61-182, builder/builder/main.py:59-65

Supported Adapter Types

The system supports several specialized adapters to handle different vendor distribution methods.

Adapter Type	Primary Data Source	Key Features
github_release	GitHub REST API	Supports regex asset matching, pre-releases, and tag pinning.
winget_manifest	Microsoft WinGet Repository	Parses YAML manifests for machine-scope installers and SHA256.
direct_url	HTML/Text feeds	Uses regex to extract versions from vendor release pages.
mozilla	JSON API	Specifically formatted for Firefox and Thunderbird version feeds.
microsoft	Visual Studio Code API	Fetches stable update metadata including hashes for VS Code.
vendor_json_api	Generic JSON	Navigates dotted JSON paths to find version and download fields.
adoptium	Adoptium API	Specialized JSON adapter for OpenJDK (Temurin) releases.

Sources: README.md:151-171, builder/builder/sources/adapters.py:64-182

Resolution Logic

GitHub Release Adapter

The github_release adapter queries the GitHub API (limiting to the last 30 releases) to find the newest stable release. It filters assets using an asset_regex and can optionally verify SHA256 hashes from a separate checksum asset.

# builder/builder/sources/adapters.py:12-45
def select_github(releases, source):
    candidates = [
        r
        for r in releases
        if not r.get("draft") and (source.get("prerelease", False) or not r.get("prerelease"))
    ]
    # ... logic to filter by tag and match asset_regex ...
    _, version, asset = max(parsed, key=lambda x: x[0])
    return Release(
        version,
        asset["browser_download_url"],
        digest[7:] if digest.startswith("sha256:") else None,
        filename=asset["name"],
    )

Sources: builder/builder/sources/adapters.py:12-58, README.md:173-189

WinGet Manifest Adapter

This adapter navigates the official Microsoft WinGet manifests stored on GitHub. It selects the highest stable version and identifies the installer that matches the package architecture (defaulting to x64) and machine scope.

The sequence shows the two-step process of identifying a version via API and retrieving specific metadata via raw file access.
Sources: builder/builder/sources/adapters.py:133-179, README.md:167-171

Direct URL and Directory Scrapers

The direct_url, videolan, and documentfoundation adapters parse text or HTML directory listings.

1. The adapter fetches a version_url.
2. It applies a version_regex to extract all available versions.
3. It sorts versions using version_key and selects the maximum.
4. It formats a download_url_template using the discovered {version}.

Sources: builder/builder/sources/adapters.py:112-132, README.md:191-196

Security and Validation

Every adapter must perform URL and host validation before returning a release to the builder.

• Hostname Pinning: The adapter validates the final download URL against an allowed_hosts list. If the vendor redirects to an untrusted host, the resolution fails unless disable_host_validation is set.
• Checksum Verification: If a checksum_url_template is provided in the catalog, the adapter fetches the checksum file and extracts the specific hash for the selected installer. This must match any hash provided by the metadata API (e.g., GitHub's internal digest).
• Credential-Free HTTPS: Only HTTPS URLs without embedded credentials, usernames, or non-standard ports are accepted.

Sources: builder/builder/sources/adapters.py:183-207, builder/builder/download.py:15-32

Configuration Summary

Discovery behavior is controlled by the source block in catalog/packages.yaml.

Parameter	Type	Description
type	String	The adapter identifier (e.g., github_release, direct_url).
allowed_hosts	List	Required list of official hostnames for installer downloads.
version_regex	String	Python regex with group 1 capturing the version string.
download_url_template	String	Template using {version}, {architecture}, or {version_compact}.
track	String	Optional major/minor filter (e.g., 3.14 for Python).
checksum_url_template	String	Optional URL to a SHA256SUMS style file.

Sources: catalog/packages.yaml:5-30, README.md:118-142, builder/builder/sources/adapters.py:112-132

Upstream Discovery Adapters ensure the repository remains current by automating the transition from vendor release feeds to actionable installer metadata while maintaining strict security boundaries via hostname allowlisting and checksum enforcement.