Creating Custom Source Adapters

Relevant source files

The following files were used as context for generating this wiki page:

• builder/builder/sources/adapters.py
• builder/builder/main.py
• builder/builder/download.py
• catalog/packages.yaml
• tests/test_workflow.py
• builder/builder/opsi.py

Source adapters in the auto-opsi-repo project resolve upstream software metadata into actionable download information. They extract version strings, installer URLs, and checksums from various vendor formats such as JSON APIs, HTML release pages, and GitHub repositories.

The system uses a dispatch mechanism in builder/builder/sources/adapters.py to select the correct logic based on the type field defined in the catalog/packages.yaml file. Each adapter must return a Release object containing the normalized version, download URL, and optional SHA256 hash.

Sources: builder/builder/sources/adapters.py:65-68, README.md:104-108

Source Architecture and Data Flow

The resolution process starts in the main execution loop, which iterates through packages defined in the catalog. The resolve function acts as the entry point, taking the package configuration and an HTTP client as arguments.

The diagram shows how the resolve function dispatches package metadata to specific adapter logic before performing final URL validation.

Key Data Structures

Component	Type	Description
Release	Dataclass	Holds version, url, sha256, filename, and detection_version.
package	Dict	The raw YAML configuration from the catalog for a specific product.
http	Object	An instance of the HTTP class providing retries and rate-limit handling.

Sources: builder/builder/sources/adapters.py:65-156, builder/builder/models.py

Implementation Requirements

To create a new source adapter, you must implement logic within builder/builder/sources/adapters.py that handles the specific metadata format of the vendor.

1. Version Discovery

Adapters must identify the latest stable version. For HTML or text-based feeds, the system utilizes version_regex to extract strings. For JSON APIs, a dot-notated field helper navigates nested structures.

def field(data, path):
    for part in path.split("."):
        data = data[int(part)] if isinstance(data, list) else data[part]
    return data

Sources: builder/builder/sources/adapters.py:59-62, builder/builder/sources/adapters.py:116-120

2. URL and Checksum Resolution

The adapter must construct a full HTTPS URL for the installer. If the vendor provides sidecar checksum files, the adapter should fetch them and extract the hash matching the selected filename.

• GitHub: Uses asset_regex to find the correct file among release assets.
• Direct URL: Uses download_url_template with placeholders like {version} and {architecture}.
• WinGet: Parses Microsoft's winget-pkgs repository to find installers matching the x64 architecture and machine scope.

Sources: builder/builder/sources/adapters.py:10-38, builder/builder/sources/adapters.py:133-162

3. URL Validation

Every resolved URL must pass through validate_url. This function enforces:

• The use of the https scheme.
• No embedded credentials (username/password).
• Hostname pinning against the allowed_hosts list defined in the catalog.

Sources: builder/builder/download.py:17-31, builder/builder/sources/adapters.py:163-164

Integrated Adapters Comparison

The following table summarizes the behavior of currently implemented source types.

Type	Metadata Source	Versioning Logic	Checksum Support
github_release	GitHub API	Tag regex / Latest flag	sha256: prefix or sidecar asset
direct_url	HTML/Text Feed	Regex max version	sha256_url_template
winget_manifest	GitHub Raw	SemVer from directory names	InstallerSha256 field
mozilla	Mozilla JSON API	Field mapping	checksum_url_template
adoptium	Adoptium API	JSON Path	sha256_field

Sources: builder/builder/sources/adapters.py:72-156, catalog/packages.yaml

Testing Custom Adapters

When adding a new adapter, you must add corresponding unit tests in tests/test_workflow.py to mock network responses and verify the resolution logic.

1. Mock HTTP Responses: Create a mock object that implements get or json methods to return sample vendor metadata.
2. Verify Release Object: Assert that the returned version, url, and sha256 match the expected values from the mock data.
3. Validate Constraints: Test that the adapter raises ValueError when metadata is ambiguous or no matching architecture is found.

Sources: tests/test_workflow.py:72-132

Summary

Creating a custom source adapter involves defining a new kind in the resolve function, implementing the logic to parse vendor-specific release data, and ensuring the final output is a validated Release object. This architecture decouples the product definitions in the catalog from the Python implementation of metadata protocols.

Sources: builder/builder/sources/adapters.py:65-70, README.md:104-108