Persistent State Management
Relevant source files
The following files were used as context for generating this wiki page:
Persistent State Management tracks the history of software versions, installer checksums, and package provenance within the auto-opsi-repo project. It ensures that the repository remains consistent across build cycles and prevents unauthorized changes to known installers.
The system stores information in the state/ directory, primarily using packages.json to record metadata for each product. This state allows the builder to detect new upstream releases, skip unchanged products, and enforce checksum immutability for existing versions.
Sources: README.md:14-15, agents.md:46-46, builder/builder/main.py:28-32
State Data Structures
The system uses specific JSON structures to persist information between runs.
packages.json Schema
The primary state file, state/packages.json, contains the following top-level keys:
| Field | Type | Description |
|---|---|---|
schema_version | Integer | The version of the state file format. |
packages | Object | Map of OPSI product IDs to their latest successful build provenance. |
checksums | Object | Map of product_id@version to the verified installer SHA256 hash. |
checksum_overrides | Object | History of accepted checksum changes per version. |
Sources: builder/builder/state.py:10-25, builder/builder/main.py:113-120
Package Provenance
Every successful build generates a provenance record. This record is stored in packages.json and also as a .provenance.json sidecar for every .opsi file in the repository.
{
"product_id": "auto-firefox",
"vendor": "Mozilla Firefox",
"upstream_version": "121.0",
"opsi_version": "121.0",
"package_revision": 1,
"sha256": "...",
"size": 1234567,
"download_timestamp": "2023-12-01T12:00:00Z",
"signature_status": "SIGNED"
}
Sources: builder/builder/main.py:90-106, builder/builder/repository.py:46-46
State Operations and Flow
The builder manages state through initialization, validation, and atomic updates.
Initialization and Recovery
When the builder starts, it reads the current state from disk. If the state file is missing or corrupted, the builder attempts to reconstruct checksum evidence by scanning all .opsi.provenance.json sidecar files in the repository. This ensures that historical checksums are protected even after state loss.
The reconstruction logic ensures that state["checksums"] is populated with existing evidence before any new downloads occur.
Sources: builder/builder/main.py:38-48, builder/builder/main.py:157-160
Checksum Immutability
The system enforces installer immutability. If an upstream version is already known in the state, the builder compares the new download's hash against the stored value.
- If hashes match, the build continues.
- If hashes differ, the builder raises a
ValueErrorand stops the build for that product. - An override is possible only if the environment variable
ALLOW_CHANGED_CHECKSUMis set totrue.
Sources: builder/builder/download.py:102-108, builder/builder/main.py:83-85, README.md:144-152
Atomic Updates
To prevent data corruption, the system writes state changes using an atomic rename pattern. The data is first written to a temporary file, which is then renamed to the final destination.
Sources: builder/builder/state.py:28-35, builder/builder/repository.py:59-65
Synchronization and Locking
The system ensures that only one builder process modifies the state at any given time.
File Locking
The builder uses state/builder.lock with fcntl.flock to serialize access. If another builder is active, the process logs an error and exits with code 2.
Process Flow
The following sequence shows how the builder interacts with the persistent state during a cycle:
Summary of Persistent Files
| File Path | Purpose | Persistence |
|---|---|---|
state/packages.json | Master record of product versions and hashes. | Persistent |
state/last-run.json | Summary of the most recent build cycle results. | Transient/Overwritten |
state/builder.lock | Prevents concurrent builder executions. | Persistent |
repository/*.provenance.json | Per-package metadata for recovery and auditing. | Persistent |
Sources: builder/builder/main.py:141-141, agents.md:46-46, README.md:88-88
Persistent State Management provides the necessary consistency for automated repository operations. By combining a centralized state file with distributed provenance sidecars and strict checksum validation, the system maintains repository integrity and prevents silent installer changes.