Artifact Manifest & Materialization
Relevant source files
The following files were used as context for generating this wiki page:
The Artifact Manifest & Materialization system serves as the portable data contract for the regime detection engine. It ensures that external market, macro, and event data—stored as durable artifacts in object storage (e.g., S3)—can be consistently reproduced and verified across different environments (local checkouts, CI, or Conductor workspaces). The system decouples the runtime engine from direct storage access by using a manifest lockfile that tracks content via SHA-256 hashes and facilitates the "materialization" of these artifacts into a local cache.
Sources: README.md:14-22, src/regime_data_fetch/materialization.py:122-132
Manifest Architecture
The manifest is a structured YAML lockfile representing a specific artifact_set. It contains metadata about the storage root and a collection of individual artifacts.
Data Models
The system uses two primary data structures defined as frozen dataclasses to ensure immutability once loaded.
| Class | Description | Key Fields |
|---|---|---|
ManifestArtifact | Represents a single data file or partitioned dataset. | name, stage, uri, local_path, sha256, required_for |
ArtifactManifest | A collection of artifacts with global metadata. | artifact_set, created_at_utc, storage_root, artifacts |
Sources: src/regime_data_fetch/artifact_manifest.py:38-51, src/regime_data_fetch/artifact_manifest.py:108-112
Artifact Stages
Artifacts transition through several stages during the data lifecycle:
raw_capture: Initial external data download.normalized: Data processed into a standard internal format.canonical: Verified, stable data ready for engine consumption.run_inputs: Specific artifacts formatted for immediate runner execution.
Sources: src/regime_data_fetch/artifact_manifest.py:9-12
Materialization Logic
Materialization is the process of synchronizing a local directory with the state defined in a manifest. The process is designed to be atomic and idempotent.
Verification and Staging
Before promoting files to the final local destination, the system performs several safety checks:
- Cache Hit Check: If a file already exists at the
local_pathand its SHA-256 matches the manifest, it is skipped. - Staging: New or drifted artifacts are downloaded from the
ArtifactStoreinto a temporary staging directory. - Integrity Validation: The system verifies the SHA-256 of the downloaded bytes against the manifest before promotion.
Atomic Promotion
Once all required artifacts are verified in the staging area, they are promoted to the final local_root. If any failure occurs during download or verification, the existing local files remain untouched, preventing a mixed or corrupt data state.
Sources: src/regime_data_fetch/materialization.py:64-106
Materialization Flow
The following diagram illustrates the lifecycle of a materialization request:
The materialization process ensures that the local data tree exactly matches the manifest's expected state.
Sources: src/regime_data_fetch/materialization.py:38-115
Manifest Resolution and Input Routing
Runners (like profile_engine.py) use a resolution layer to map logical input fields to physical file paths materialized from the manifest.
Input Specification Registry
The MANIFEST_INPUT_SPECS registry acts as the single source of truth for routing. It maps engine fields (e.g., macro_parquet) to artifact names (e.g., fred_macro_series).
| Field | CLI Flag | Artifact Name | Required |
|---|---|---|---|
macro_parquet | --macro-parquet | fred_macro_series | Yes |
pit_parquet | --pit-parquet | sp500_pit_constituents | Yes |
event_calendar | --event-calendar | event_calendar_us | Yes |
pmi_path | --pmi-path | ism_pmi_history | No |
Sources: src/regime_data_fetch/manifest_inputs.py:108-144
Path Resolution Sequence
The resolver identifies paths using a hierarchy of priorities:
- CLI Overrides: Manual flags passed to the script.
- Manifest Artifacts: Lookups based on the
required_fortag in the manifest. - Default Relpaths: Conventional locations under
data/raw/used if no manifest is provided.
Sources: src/regime_data_fetch/manifest_inputs.py:210-244, README.md:38-46
Canonicalization and Publishing
To maintain byte-level stability across machines, the system employs a "canonicalization" step for Parquet files before they are written to the manifest.
Parquet Stability
The publish_canonical_snapshot.py script enforces stable Parquet bytes by:
- Sorting rows ascending by all sortable columns.
- Stripping schema metadata (avoiding pandas-specific footer noise).
- Pinning compression to
snappyand timestamp coercion tous. - Ensuring
canonicalize(canonicalize(x)) == canonicalize(x).
Sources: scripts/publish_canonical_snapshot.py:92-104, scripts/publish_canonical_snapshot.py:113-134
Semantic Validation
During publishing, artifacts undergo semantic checks to ensure they meet the data contract:
- OHLCV Contract: Checks for required columns (
open,high,low,close,volume,symbol) and validates date coverage against the SPY trading calendar. - Metadata Drift: Compares actual row counts and date ranges (
min_date,max_date) against manifest entries.
Sources: scripts/publish_canonical_snapshot.py:171-200, scripts/publish_canonical_snapshot.py:284-306
Command Line Interface
The materialization and publishing workflows are exposed via dedicated CLI scripts.
Materialization CLI (materialize_regime_data.py)
This script allows manual synchronization of the local data root.
python3 scripts/materialize_regime_data.py \
--manifest manifests/runs/regime_engine_2026-05-17.yaml \
--local-root data/raw \
--required-for profile_engine
Sources: scripts/materialize_regime_data.py:22-45
Publishing CLI (publish_canonical_snapshot.py)
Used to update the manifest and storage root after a data fetch.
--check: Recomputes SHAs in memory and exits with error on drift.--dry-run: Reports which artifacts would be uploaded or canonicalized.publish(default): Canonicalizes on-disk bytes, uploads to S3, and rewrites the manifest metadata.
Sources: scripts/publish_canonical_snapshot.py:15-25, scripts/publish_canonical_snapshot.py:488-510
Summary
Artifact Manifest & Materialization provides the regime-detection project with a deterministic data supply chain. By tracking every input file with SHA-256 hashes and managing local caches through an atomic staging process, the system guarantees that analysis results are reproducible regardless of the environment in which the engine is executed. This architecture supports complex data dependencies, such as symbol-partitioned OHLCV trees, while maintaining a simple interface for operators through manifest-routed inputs.