Local History Recovery
Relevant source files
The following files were used as context for generating this wiki page:
Local History Recovery restores missing market data and artifacts in the Nexus Scanner environment. It synchronizes local caches with external source checkouts or official exchange archives to ensure the technical indicator engine has sufficient lookback periods for accurate screening. This system addresses gaps caused by GitHub Actions cache evictions or incomplete daily refreshes.
Sources: frontend/README.md:387-393, DO NOT DELETE EDL PIPELINE/README.md:88-93
Architecture and Data Flow
The recovery process integrates multiple data sources into the local ohlcv_data/ and delivery_history_data/ directories. It prioritizes official NSE data for recent sessions and utilizes EOD2 checkouts for longer adjusted historical seeding.
Core Recovery Logic
The sync_local_scanner_history.py tool serves as the primary interface for manual restoration. It merges history from a source root, applies official daily data for a specified range, and generates a local_scanner_history_report.json.
The diagram shows the logic branches between merging existing local data and fetching missing records from exchange archives.
Sources: frontend/README.md:395-408, DO NOT DELETE EDL PIPELINE/apply_nse_daily_ohlcv.py:36-58
Official NSE Gap Recovery
The pipeline automatically attempts to recover missing candles using official NSE full bhavcopy files. This mechanism targets gaps outside of known adjusted history.
- Boundary Protection: Raw prices are only accepted after a symbol's adjusted EOD2 history boundary or into an empty cache to prevent mixing price bases.
- Fallback Mechanism: Gaps within adjusted history or I/O failures during recovery fall through to provider-specific synchronization.
- Validation: Recovered candles undergo completeness checks and fast ISO parsing before being merged into local CSV files.
Sources: DO NOT DELETE EDL PIPELINE/README.md:88-94, DO NOT DELETE EDL PIPELINE/apply_nse_daily_ohlcv.py:36-47
Gap Recovery Sequence
This sequence illustrates how the pipeline identifies missing sessions and fills them using official exchange data.
Sources: DO NOT DELETE EDL PIPELINE/apply_nse_daily_ohlcv.py:36-62, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:345-350
EOD2 Historical Bootstrap
For seeding long-term adjusted history, the pipeline utilizes local checkouts of EOD2 data. This is typically invoked during weekly refreshes or initial environment setup.
| Feature | Description |
|---|---|
| ISIN Matching | Joins renamed segments by ISIN rather than ticker filename. |
| Price Policy | Supplies split and bonus-adjusted OHLC data. |
| Volume Policy | Retains actual historical traded quantity (unadjusted). |
| Overlay Logic | Retains local candles newer than the EOD2 snapshot. |
Sources: DO NOT DELETE EDL PIPELINE/README.md:98-115, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:352-355
Component Roles
Several scripts and utilities contribute to the recovery and synchronization of local history:
import_eod2_ohlcv.py: Handles the ingestion of adjusted historical data from git-cloned data repositories.apply_nse_daily_ohlcv.py: Merges the latest closed session data from the NSE into the local cache.sync_local_scanner_history.py: Coordinates manual synchronization between different EDL installations.ohlcv_utils.py: Provides shared helpers for candle parsing, CSV I/O, and date merging.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:100-110, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:200-205
Recovery Procedures
To perform a manual history recovery, follow these steps:
- Activate the Python virtual environment in the pipeline directory.
- Identify a compatible source EDL directory containing the desired history.
- Run the synchronization script with the appropriate date range:
python3 sync_local_scanner_history.py \ --source-root /path/to/existing/EDL \ --from-date 2026-09-29 \ --as-of-date 2026-09-30 - Verify the
local_scanner_history_report.jsonfor merge status and ISIN mismatches. - Rebuild chart artifacts and publish a new snapshot to reflect the recovered data in the frontend.
Sources: frontend/README.md:395-415, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py:115-125
Summary
Local History Recovery is critical for maintaining the integrity of the Nexus Scanner technical engine. By combining official exchange archives for recent daily gaps and EOD2 checkouts for long-term adjusted history, the system ensures that moving averages (e.g., SMA 200) and multi-year performance metrics remain accurate across all supported securities.