NSE & SME Data Acquisition

Relevant source files

The following files were used as context for generating this wiki page:

• DO NOT DELETE EDL PIPELINE/fetch_sme_data.py
• DO NOT DELETE EDL PIPELINE/filter_mainboard_universe.py
• DO NOT DELETE EDL PIPELINE/reconcile_nse_equity_universe.py
• DO NOT DELETE EDL PIPELINE/apply_nse_daily_ohlcv.py
• DO NOT DELETE EDL PIPELINE/README.md
• DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py

NSE & SME Data Acquisition refers to the foundational processes within the EDL Pipeline that fetch, validate, and reconcile security data from the National Stock Exchange (NSE) and Small and Medium Enterprise (SME) market feeds. This system ensures the project's scanner universe is limited to verified Mainboard EQ securities by filtering out SME listings and unlisted stocks.

The system acquires official NSE equity lists, delivery reports, and SME market watch data to establish a canonical universe for downstream analysis scripts like fundamentals, OHLCV, and market breadth. It serves as a quality gate, preventing the publication of artifacts based on stale or truncated listing responses.

Universe Selection and Filtering

The pipeline filters the initial provider data (Dhan/ScanX) to create a Mainboard-only universe. It uses the official NSE SME market-watch feed to identify and remove SME symbols from the master ISIN map.

Mainboard Filtering Logic

The filtering process involves three primary steps:

1. SME Exclusion: It removes any symbol found in the sme_market_data.json feed from the master_isin_map.json.
2. NSE Reconciliation: It requires symbols to exist in the freshly validated NSE EQUITY_L.csv list.
3. Session Alignment: It defers listings that occurred after the latest completed NSE bhavcopy session date.

Sources: README.md:126-132, filter_mainboard_universe.py:10-15

Data Flow for Universe Filtering

The diagram shows how three distinct sources are merged and filtered to produce the final mainboard scan data used by the scanner.
Sources: filter_mainboard_universe.py:90-115

SME Data Acquisition

The SME data fetcher retrieves the current NSE SME market watch data. This data is critical for classifying and subsequently excluding SME securities from the primary scanner universe.

SME Fetch Parameters

Parameter	Value	Description
API URL	https://www.nseindia.com/api/NextApi/apiClient/marketWatchApi?functionName=getSmeData	The endpoint for SME market data.
Referer	https://www.nseindia.com/market-data/sme-market	Required header to bypass NSE anti-bot triggers.
Output File	sme_market_data.json	JSON storage for cleaned SME records.

Sources: fetch_sme_data.py:10-14, README.md:126-128

NSE Equity Reconciliation

The reconciliation process identifies NSE-listed EQ series securities that are missing from the provider (ScanX) data. It categorizes these securities as either "pending" or "alerts" based on the duration of their absence.

Reconciliation Alerts

If an NSE-listed security remains absent from the provider feed after two observed weekday sessions, the system marks it as an alert in nse_universe_reconciliation.json. Rights and non-EQ series are reported separately and are never treated as IPO candidates.

Sources: reconcile_nse_equity_universe.py:16-18, README.md:144-148

Reconcile Procedure

1. Load the master_isin_map.json and the NSE EQUITY_L.csv.
2. Identify symbols in the NSE list that do not exist in the master map.
3. Filter out non-EQ series.
4. Calculate observed_sessions_since_listing by counting weekdays between the listing date and the current session.
5. Set status to alert if the count is greater than or equal to 2.

Sources: reconcile_nse_equity_universe.py:53-75

Official NSE OHLCV Application

The pipeline applies official NSE full bhavcopy candles to the local OHLCV cache. This ensures the scanner uses exchange-verified prices for the latest closed session.

Sequence of OHLCV Updates

The sequence illustrates that the official NSE close takes precedence over other price sources for a specific session date.
Sources: apply_nse_daily_ohlcv.py:22-38, src/edl_pipeline/runner.py:382-390

Pipeline Integrity Gates

Acquisition scripts include specific failure conditions to protect the integrity of the data universe. A reconciliation that would exclude more than 5% of the pre-reconciliation mainboard universe triggers a pipeline failure to guard against truncated listing responses.

Critical Validation Rules

• Freshness: Universe filtering requires a freshly fetched completed NSE session. Future session dates are rejected.
• Mandatory Sources: If fetch_dhan_data.py or fetch_sme_data.py fails, the pipeline stops immediately because downstream scripts cannot safely determine the universe.
• Schema Consistency: All output artifacts must match the REQUIRED_FINAL_FIELDS defined in the scanner schema before publication.

Sources: filter_mainboard_universe.py:73-82, src/edl_pipeline/runner.py:348-355, src/edl_pipeline/artifacts.py:5-10

Summary of Acquisition Components

Script	Responsibility	Primary Artifact
fetch_sme_data.py	Retrieves NSE SME market watch feed	sme_market_data.json
filter_mainboard_universe.py	Removes SME and unlisted symbols	mainboard_scanx_data.json
reconcile_nse_equity_universe.py	Compares NSE listings vs ScanX coverage	nse_universe_reconciliation.json
apply_nse_daily_ohlcv.py	Injects official NSE session into cache	ohlcv_data/{SYMBOL}.csv
fetch_nse_delivery_data.py	Fetches session bhavcopy and delivery	nse_delivery_data.json

Sources: README.md:270-285, src/edl_pipeline/artifacts.py:100-115

NSE & SME Data Acquisition provides the structural foundation for the EDL Pipeline. By centralizing the logic for universe filtering and official price application, the system ensures that all subsequent analysis, from market breadth to fundamental screening, operates on a validated and accurate set of securities.