Introduction & Overview
Relevant source files
The following files were used as context for generating this wiki page:
Nexus Scanner is an NSE (National Stock Exchange) equity research application that integrates an interactive frontend screener with a Python-based Extract-Deliver-Load (EDL) pipeline. The system provides tools for technical and fundamental analysis, IPO tracking, and market breadth research using immutable data snapshots generated from official NSE and Dhan ScanX sources.
The project architecture separates data acquisition and artifact generation (handled by the Python EDL pipeline) from the user interface and runtime evaluation (handled by the React frontend). This ensures that researchers can work with validated, session-pinned data while maintaining high performance through browser-side decompression and evaluation of precomputed metrics.
Sources: frontend/README.md:1-12, DO NOT DELETE EDL PIPELINE/README.md:1-10
System Architecture
The Nexus Scanner architecture consists of a backend data pipeline and a frontend runtime environment. The pipeline fetches raw data from multiple providers, validates it, and publishes immutable revisions. The frontend consumes these revisions and performs either browser-side or server-side (Python) evaluation based on query complexity.
Data Flow and Pipeline Phases
The EDL pipeline operates in six distinct phases to ensure data integrity and proper dependency management.
The diagram illustrates the sequential phases of the EDL pipeline from raw data acquisition to final compressed artifacts.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:270-380, DO NOT DELETE EDL PIPELINE/README.md:143-151
Frontend Runtime
At runtime, the frontend application follows a strict lifecycle to ensure the user is viewing the latest published trading session.
- Release Resolution: The app fetches
current.jsonto identify the active revision and session. - Snapshot Loading: A Web Worker fetches and decompresses the
stocks.json.gzsnapshot. - Evaluation:
- Supported conditions (e.g., RVOL, SMA) are evaluated directly in the browser.
- Unsupported expressions (historical text queries) are sent to a Python bridge via
POST /api/screens/run.
- Verification: The adapter rejects any response that does not match the pinned immutable revision.
Sources: frontend/README.md:209-222, frontend/README.md:258-270
Key Components & Components
EDL Pipeline Modules
The pipeline is managed by run_full_pipeline.py and structured as follows:
| Component | Responsibility | Relevant Files |
|---|---|---|
| Core Fetchers | Fetches stock data, ISIN maps, and SME lists from Dhan ScanX. | fetch_dhan_data.py, fetch_sme_data.py |
| Universe Filter | Reconciles provider data with official NSE equity lists to ensure a mainboard-only universe. | filter_mainboard_universe.py |
| OHLCV Engine | Manages incremental daily and weekly adjusted OHLCV history. | fetch_all_ohlcv.py, import_eod2_ohlcv.py |
| Metrics Processor | Injects calculated metrics like ADR, RVOL, and All-Time Highs. | advanced_metrics_processor.py |
| Breadth Engine | Generates MBI/XP market-breadth artifacts and sector analytics. | process_mbi_market_breadth.py |
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:65-125, DO NOT DELETE EDL PIPELINE/README.md:221-255
Frontend API Facade
The screenerApi acts as a typed facade between the UI and the data adapters.
The sequence shows how queries are routed between browser-side evaluation and the Python historical evaluator.
Sources: frontend/README.md:258-270, frontend/README.md:280-286
Data Artifacts
The system relies on immutable, content-addressed files. Each successful pipeline run produces a set of artifacts required for the frontend to function.
| Artifact | Format | Description |
|---|---|---|
stocks.json.gz | Gzip JSON | The primary snapshot containing all calculated technical and fundamental metrics for the session. |
release.json | JSON | Immutable metadata describing the session date and artifact pointers. |
market_breadth_v2.json.gz | Gzip JSON | Contains processed MBI/XP breadth records and quality diagnostics. |
corporate_action_ledger.json.gz | Gzip JSON | A record of splits, bonuses, and dividends with adjustment factors. |
ipo_screener.json.gz | Gzip JSON | The IPO catalogue with listing windows and provider data. |
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:40-60, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:480-510
Validation & Reliability
The pipeline implements rigorous validation gates to prevent the publication of corrupt or stale data.
- Exit Codes: Critical foundation scripts (e.g.,
fetch_dhan_data.py) must exit with code 0. Failure stops the pipeline immediately. - Artifact Specs: The
ArtifactSpecclass defines required fields, minimum record counts, and format (JSON, CSV, Gzip). - Universe Reconciliation: A refresh fails if reconciliation would exclude more than 5% of the pre-reconciliation mainboard universe, protecting against truncated provider responses.
- Atomic Writes: Final artifacts use atomic JSON and Gzip writes to ensure interrupted processes do not leave partial files.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:40-55, DO NOT DELETE EDL PIPELINE/filter_mainboard_universe.py:80-95, DO NOT DELETE EDL PIPELINE/README.md:153-170
Summary
Nexus Scanner provides a robust framework for NSE research by coupling a high-concurrency Python data pipeline with a React-based interactive UI. By utilizing immutable revisions and a hybrid evaluation engine, the system ensures analysts work with consistent, validated data across both simple technical filters and complex historical queries. Significant emphasis is placed on pipeline reliability through multi-phase execution, strict artifact validation, and universe reconciliation against official exchange records.