Retention Policy and Cleanup
Relevant source files
The following files were used as context for generating this wiki page:
The Retention Policy and Cleanup system manages the lifecycle of .opsi packages and temporary build artifacts. It ensures the repository remains within storage limits by removing obsolete versions while maintaining metadata integrity.
The repo-builder component executes cleanup logic at the end of every successful build cycle. It identifies obsolete archives based on a configurable version count and removes them alongside their associated sidecar files. This process guarantees that the newest software releases are always available for OPSI clients while preventing disk exhaustion.
Sources: README.md:14-18, builder/builder/main.py:150-155
Version Retention Policy
The system retains a specific number of complete product versions and revisions. By default, it keeps the two newest versions per product. The policy always protects the newest archive and only removes older files after a new repository metadata generation succeeds.
Sources: README.md:158-161, builder/builder/repository.py:33-35
Configuration Options
Retention behavior is controlled primarily through environment variables defined in the Docker configuration.
| Variable | Default | Description |
|---|---|---|
KEEP_VERSIONS | 2 | Number of versions/revisions to retain per product. Must be at least 1. |
REPO_ROOT | /data | Root directory containing repository, state, and work folders. |
Sources: README.md:58-61, docker-compose.yml:19-20, builder/builder/main.py:36-38
Obsolescence Logic
The system identifies obsolete packages by scanning the repository directory for .opsi files. It parses filenames to extract the product ID, version, and revision. Packages are grouped by product ID and sorted by version and revision to determine the retention order.
The inventory function uses a regular expression to validate and extract metadata from filenames like auto-firefox_143.0-1.opsi.
Sources: builder/builder/repository.py:10-22, builder/builder/repository.py:25-31, tests/test_core.py:141-150
Cleanup Process
The cleanup process follows a strict sequence to prevent repository corruption. It generates new metadata excluding obsolete files before physically deleting any data.
Metadata Synchronization
Before deletion, the metadata function creates a temporary staging directory in the work folder. It links or copies all non-obsolete .opsi files into this stage and invokes opsi-cli to generate fresh metadata files (packages.json, packages.msgpack.zstd). These new metadata files are then atomically moved into the live repository.
Sources: builder/builder/repository.py:38-57, README.md:21-25
File Deletion
Once metadata is updated, the system removes the obsolete .opsi archives. It also deletes all related sidecar files to ensure a clean state.
.opsi(Package archive).md5(Checksum for OPSI compatibility).zsync(Delta download metadata).provenance.json(Internal metadata and checksum history)
Sources: builder/builder/repository.py:64-68, tests/test_workflow.py:270-281
Temporary Workspace Cleanup
The work/ directory stores temporary downloads, extracted payloads, and metadata staging areas. The builder performs a "crash-leftover" cleanup at the start of every cycle.
Startup Cleanup Procedure
When the builder starts under a file lock, it iterates through the work directory and removes directories matching specific patterns:
auto-*: Temporary product build directories.metadata-*: Metadata generation stages.ziplaunch.*.opsi-cli.*: Temporary OPSI CLI runtime caches.
Integrity Guarantees
The retention system includes safeguards to ensure that network failures or builder crashes do not leave the repository in an inconsistent state.
- Metadata First: Metadata generation must succeed before any deletions occur. A failure in metadata generation stops the retention process, leaving old packages available.
- Atomic Replacement: Metadata files and new packages are renamed atomically from hidden temporary files (e.g.,
.packages.json.tmp) to their final names. - Provenance Persistence: Historical checksums in
state/packages.jsonare preserved even after archives are pruned to prevent "changed checksum" attacks on future updates.
Sources: builder/builder/repository.py:58-61, builder/builder/repository.py:75-84, README.md:25-27, README.md:143-145
Summary
Retention and cleanup in the auto-opsi-repo project is an automated, metadata-driven process. By combining version-based pruning with atomic file operations and startup workspace purging, the system maintains a slim, high-integrity repository suitable for production OPSI environments.