ScanX & Dhan Broker Integrations
Relevant source files
The following files were used as context for generating this wiki page:
- DO NOT DELETE EDL PIPELINE/README.md
- DO NOT DELETE EDL PIPELINE/fetch_dhan_data.py
- DO NOT DELETE EDL PIPELINE/fetch_fundamental_data.py
- DO NOT DELETE EDL PIPELINE/fetch_scanx_ipo_data.py
- DO NOT DELETE EDL PIPELINE/fetch_circuit_stocks.py
- DO NOT DELETE EDL PIPELINE/enrich_fno_data.py
- frontend/README.md
ScanX and Dhan Broker integrations provide the primary data acquisition layer for the EDL (Extract, Data, Load) pipeline. This system retrieves market quotes, fundamental financial statements, corporate actions, and technical indicators from various Dhan and ScanX unofficial endpoints. These integrations facilitate the creation of a comprehensive equity research database for the Indian National Stock Exchange (NSE).
The pipeline uses these integrations to fetch data for approximately 2,775 stocks, covering everything from real-time news and announcements to historical OHLCV (Open, High, Low, Close, Volume) data. The system is designed to be resilient, employing bounded retries, exponential backoff, and atomic writes to ensure data integrity during the acquisition process.
Sources: DO NOT DELETE EDL PIPELINE/README.md:1-25, frontend/README.md:1-20
Data Acquisition Architecture
The architecture relies on a multi-lane fetch strategy to avoid serial bottlenecks. Core data serves as the foundation for enrichment lanes, which proceed in parallel before the final analysis and build phases.
Pipeline Flow Diagram
The following diagram illustrates the sequence of data acquisition and processing from Dhan and ScanX sources.
Primary Integration Endpoints
The project integrates with several distinct API families provided by Dhan and ScanX. Each serves a specific data requirement, ranging from static fundamental snapshots to streaming-style news.
Market and Fundamental APIs
| Endpoint Type | Target URL | Method | Data Provided |
|---|---|---|---|
| Full Market | https://ow-scanx-analytics.dhan.co/customscan/fetchdt | POST | NSE Total Market universe, basic quotes, market cap. |
| Fundamentals | https://open-web-scanx.dhan.co/scanx/fundamental | POST | P&L statements, Balance Sheets, Shareholding patterns. |
| Indicators | https://ow-static-scanx.dhan.co/staticscanx/indicator | POST | EMA, SMA, Pivot Points (Classic/Fibonacci). |
| Live News | https://news-live.dhan.co/v2/news/getLiveNews | POST | Real-time media feeds with sentiment analysis. |
Sources: DO NOT DELETE EDL PIPELINE/README.md:213-285, DO NOT DELETE EDL PIPELINE/fetch_fundamental_data.py:10-15
ScanX Client Sequence
The following sequence diagram shows the interaction between the local fetcher and the ScanX IPO service.
Implementation Details
Fundamental Data Fetching
The fetch_fundamental_data.py script iterates through the master_isin_map.json and requests financial data in batches. It appends financial_observed_on timestamps to ensure that historical snapshots cannot be retroactively rewritten with data seen later.
- Batch Size: 100 ISINs per request.
- Delay: 0.5 seconds between requests to avoid rate limiting.
- Error Handling: Refuses partial refreshes; all batches must succeed for the output to be saved.
Sources: DO NOT DELETE EDL PIPELINE/fetch_fundamental_data.py:17-65
IPO Enrichment Engine
The ScanX IPO integration fetches data from two primary endpoints (listedIpos and history) to build a durable archive. It uses a "slug" (SEO symbol) to retrieve specific historical details for each company.
# fetch_scanx_ipo_data.py:170-185
def _detail_body(slug: str) -> dict:
return {
"entity_id": "", "source": "O", "token_id": "", "iv": "",
"data": {"cred": "thor", "seoSymbol": slug}, "aes_key": "", "ip": "",
}
Sources: DO NOT DELETE EDL PIPELINE/fetch_scanx_ipo_data.py:170-185
Circuit and F&O Enrichment
Specific scripts target intraday market conditions and derivative segment status:
- Circuit Stocks: Uses
customscan/fetchdtwith filters forLiveData.UpperCircuitBreakandLiveData.LowerCircuitBreak. - F&O Status: Fetches lot sizes and expiry calendars from
dhan.co/_next/data/. It maps these using anormalized_symbolfunction to ensure consistency between the Dhan build IDs and NSE ticker formats.
Sources: DO NOT DELETE EDL PIPELINE/fetch_circuit_stocks.py:12-45, DO NOT DELETE EDL PIPELINE/enrich_fno_data.py:35-70
Data Standardisation
The pipeline standardises raw ScanX and Dhan responses into a "Scanner Schema v3". This involves converting raw field names into snake_case and normalizing values (e.g., converting "Yes/No" to boolean true/false).
| Raw Field | Standardized Key | Description |
|---|---|---|
Sym | symbol | Ticker symbol. |
Mcap | market_cap_crore | Market capitalization in ₹ Crore. |
PPerchange | change_percent | Percentage price change for the session. |
High1Yr | high_52w | 52-week high price. |
Sources: DO NOT DELETE EDL PIPELINE/standardize_stock_artifact.py:65-120, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/schemas.py:25-50
Conclusion
The Dhan and ScanX integrations form a robust backbone for the EDL pipeline. By leveraging a mix of direct POST APIs and Next.js data files, the system maintains a comprehensive and timely dataset of the Indian equity market. The use of immutable artifacts and validation gates ensures that the data published to the frontend scanner remains accurate and consistent across sessions.