Builder Core Application Loop

Relevant source files

The following files were used as context for generating this wiki page:

• builder/builder/main.py
• builder/builder/download.py
• builder/builder/opsi.py
• builder/builder/sources/adapters.py
• README.md
• agents.md

The Builder Core Application Loop manages the automated lifecycle of OPSI packages within the repo-builder service. It discovers upstream releases, downloads installers, builds .opsi archives, and maintains the repository state. The loop executes periodically or as a single-run task based on environment configuration.

Sources: README.md, builder/builder/main.py:126-155

Execution Model

The application loop runs inside the repo-builder container. It follows a structured sequence of initialization, locking, processing, and sleeping.

Process Synchronization and Initialization

The builder prevents concurrent writes by using a file lock (state/builder.lock) via fcntl.flock. If another builder process is active, the loop terminates to prevent data corruption. Upon obtaining the lock, the builder cleans the work/ directory to remove temporary files from previous crashed runs.

Sources: builder/builder/main.py:136-150

Periodic Schedule

When running in service mode, the builder sleeps for a duration defined by UPDATE_INTERVAL_SECONDS (default: 21600 seconds) after each cycle. A threading.Event handles termination signals (SIGTERM, SIGINT) to allow clean exits.

Sources: builder/builder/main.py:133-158, docker-compose.yml:20

Package Processing Lifecycle

The run() function iterates through every package defined in catalog/packages.yaml. For each enabled package, it performs a multi-stage synchronization process.

Sources: builder/builder/main.py:25-56

1. Metadata Resolution

The builder calls resolve() to identify the latest upstream version using various adapters (GitHub, Microsoft, Mozilla, WinGet, etc.). This stage retrieves the download URL and expected SHA256 checksum without downloading the actual payload.

Sources: builder/builder/sources/adapters.py:67-183

2. Version and Checksum Validation

The builder compares the discovered upstream version against the existing repository inventory.

• Downgrade Prevention: If the upstream version is older than the current repository version, the builder blocks the update.
• Checksum Immutability: The builder compares the new checksum against historical data in state/packages.json. If a checksum for a known version changes, the process fails unless ALLOW_CHANGED_CHECKSUM is enabled.

Sources: builder/builder/main.py:58-85, builder/builder/download.py:169-175

3. Download and Packaging

If a build is required, the builder performs the following steps:

1. Download: Retrieves the installer into a temporary directory under work/.
2. Signature Verification: Executes osslsigncode to check if the binary is signed.
3. Source Generation: Renders OPSI control files and scripts using Jinja2 templates.
4. Package Build: Calls opsi-cli package make to create the .opsi archive.
5. Validation: Extracts the generated package to verify the integrity of the archive and control metadata.

Sources: builder/builder/download.py:186-224, builder/builder/opsi.py:25-115

Repository Maintenance

After processing all individual packages, the builder performs repository-wide maintenance tasks.

Metadata Generation

The builder executes opsi-cli repository create to regenerate the packages.json and associated metadata files used by OPSI clients. This happens outside the served directory and uses atomic renames to ensure availability.

Sources: README.md, builder/builder/opsi.py:14-22

Retention Policy

The retain() function enforces the KEEP_VERSIONS limit. It identifies obsolete archives and deletes them along with their sidecar files (.md5, .zsync, .provenance.json). The builder always retains the newest version and only deletes files after successful metadata generation.

Sources: builder/builder/main.py:116-121, builder/builder/repository.py

Configuration Summary

The application loop behavior is controlled by environment variables.

Variable	Description	Default
UPDATE_INTERVAL_SECONDS	Delay between cycles in service mode	21600
KEEP_VERSIONS	Number of package versions to retain	2
ALLOW_CHANGED_CHECKSUM	Permit hash changes for known versions	false
MAX_DOWNLOAD_BYTES	Maximum allowed size for an installer	2147483648
LOG_LEVEL	Verbosity of the builder logs	INFO

Sources: README.md, docker-compose.yml:15-22

Error Handling and Isolation

The loop implements failure isolation: one product failing does not stop the processing of other products. Specific errors, such as GitHub API rate limits, are caught and logged as warnings, causing the builder to skip affected products until the next cycle. Fatal errors during repository metadata generation prevent the deletion of old packages to ensure the repository remains functional.

Sources: builder/builder/main.py:108-124, README.md

The Builder Core Application Loop ensures that the OPSI repository stays updated with minimal manual intervention while maintaining strict validation of installer integrity and version ordering.