Constituent OHLCV Tree Management

Relevant source files

The following files were used as context for generating this wiki page:

• src/regime_data_fetch/constituent_ohlcv_tree.py
• src/regime_data_fetch/pit_constituents.py
• src/regime_data_fetch/local_daily_ohlcv_sqlite.py
• scripts/materialize_constituent_ohlcv_tree.py
• tests/test_constituent_ohlcv_tree_materialization.py
• scripts/profile_engine.py

Constituent OHLCV Tree Management is a core data engineering subsystem responsible for the acquisition, organization, and materialization of historical market data (Open, High, Low, Close, Volume) for equity universes. The system ensures that the regime detection engine has access to high-fidelity, point-in-time (PIT) accurate data by managing partitioned Parquet structures—often referred to as "trees"—where data is organized by ticker symbol.

The primary objective of this module is to transform raw market data into canonical "profile trees" that align with specific point-in-time constituent lists (such as the S&P 500). This process involves resolving membership intervals, validating data integrity through SHA-256 checksums, and providing efficient access patterns for downstream feature engineering and backtesting components.

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py, scripts/materialize_constituent_ohlcv_tree.py, src/regime_data_fetch/universe.py

Architecture and Data Structures

The OHLCV Tree Structure

The project uses a directory-partitioned storage format where each ticker symbol's data is stored in a dedicated sub-directory. This format supports both single Parquet files and multi-file partitions.

Component	Path Pattern	Description
Root Tree	data/raw/daily_ohlcv	The base directory containing all symbols.
Symbol Partition	symbol=TICKER/	Directory containing data for a specific symbol.
Data File	ohlcv.parquet	The Parquet file containing OHLCV records.
Manifest	MANIFEST.sha256.json	JSON file containing file-level and aggregate checksums.

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:126-140, scripts/materialize_constituent_ohlcv_tree.py:31-36

Core Data Models

The materialization process utilizes specific data structures to track the state and integrity of the resulting tree.

@dataclass(frozen=True)
class ConstituentTreeMaterialization:
    report_path: Path
    manifest_path: Path
    output_tree: Path
    requested_symbols: int
    written_symbols: int
    missing_symbols: tuple[str, ...]
    aggregate_sha256: str

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:27-36

Materialization Logic

The materialization process creates a "canonical" tree by filtering a source tree based on Point-in-Time (PIT) constituent intervals for a specific date range.

Process Flow

The materialization logic follows a strict sequence:

1. PIT Resolution: Identifies symbols that were members of the universe at any point during the start_date to end_date window.
2. Staging: Market data is read from the source tree and written to a temporary staging directory to ensure atomicity.
3. Validation: Columns are verified against EXPECTED_OHLCV_COLUMNS and symbol consistency is checked.
4. Checksum Generation: SHA-256 digests are calculated for every file and for the aggregate tree.
5. Finalization: The staging directory replaces the target output_tree and a manifest is written.

Data Validation Requirements

Every Parquet file in the tree must contain the following columns:

• date
• symbol
• open
• high
• low
• close
• volume
• adjusted_close

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:15-24, src/regime_data_fetch/local_daily_ohlcv_sqlite.py:382-390

Point-in-Time (PIT) Integration

The system relies on PIT intervals to determine which symbols belong in a materialized tree. These intervals are typically sourced from community-maintained lists or vendor feeds.

PIT Interval Structure

Field	Type	Description
ticker	string	The ticker symbol.
start_date	date	The date the ticker entered the universe.
end_date	date	The date the ticker exited (null if still active).

Sources: src/regime_data_fetch/pit_constituents.py:41-47, src/regime_data_fetch/constituent_ohlcv_tree.py:118-125

Overlap Logic

The function _load_pit_overlap_symbols identifies symbols where:

• The interval start_date is less than or equal to the requested end_date.
• The interval end_date is either null or greater than or equal to the requested start_date.

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:130-138

Integrity and Provenance

SHA-256 Checksumming

To guarantee reproducibility, the system generates a manifest containing:

• File-level SHA-256: Calculated for each individual ohlcv.parquet file.
• Aggregate SHA-256: A digest of the sorted symbol-to-checksum mapping.

def _aggregate_sha256(file_entries: list[dict[str, object]]) -> str:
    digest = hashlib.sha256()
    for item in sorted(file_entries, key=lambda entry: str(entry["symbol"])):
        digest.update(f"{item['symbol']}|{item['sha256']}\n".encode("utf-8"))
    return digest.hexdigest()

Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:183-188

Reporting

The materialization results in a JSON report containing counts of requested vs. written symbols, a list of missing symbols, and paths to the source and output artifacts. This is used for auditing and debugging data gaps.
Sources: src/regime_data_fetch/constituent_ohlcv_tree.py:87-105

Operational Tools

materialization script

The script scripts/materialize_constituent_ohlcv_tree.py provides a command-line interface for generating trees.

Argument	Default	Description
--source-tree	data/raw/daily_ohlcv	Path to the master OHLCV tree.
--out-tree	data/raw/daily_ohlcv_762	Path to the canonical output tree.
--pit-parquet	.../sp500_ticker_intervals.parquet	PIT membership data.
--start / --end	(Required)	The date range for membership overlap.
--allow-missing-symbols	False	If true, records missing data instead of failing.

Sources: scripts/materialize_constituent_ohlcv_tree.py:27-60

SQLite Import

While Parquet is the primary storage format for trees, the system supports importing these trees into a SQLite row store for efficient random access during specific analysis tasks.

Summary

Constituent OHLCV Tree Management provides a robust, verifiable pipeline for creating point-in-time accurate market data snapshots. By leveraging partitioned Parquet storage, PIT membership intervals, and SHA-256 manifest validation, the system ensures that regime detection models are trained and executed on consistent and auditable data artifacts. This management is critical for preventing look-ahead bias and ensuring backtest integrity across shifting market universes.