System Architecture
Relevant source files
The following files were used as context for generating this wiki page:
- DO NOT DELETE EDL PIPELINE/README.md
- frontend/README.md
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py
- DO NOT DELETE EDL PIPELINE/filter_mainboard_universe.py
- DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/mbi.py
Nexus Scanner comprises two primary systems: a Python-based EDL (Extract, Delivery, Load) Pipeline and a React-based Frontend Application. The pipeline retrieves raw market data from public and unofficial NSE and Dhan endpoints, standardizes it, and generates immutable JSON artifacts. The frontend consumes these artifacts to provide a research interface for NSE equities, utilizing a browser-side Web Worker engine for real-time screening and a Python bridge for complex historical queries.
Sources: frontend/README.md:5-10, DO NOT DELETE EDL PIPELINE/README.md:1-5
Data Pipeline Architecture
The EDL Pipeline operates as a multi-phase orchestrator that transforms raw exchange data into validated scanner artifacts. It uses a master runner to execute scripts in dependency order, ensuring data integrity through mandatory validation gates.
Pipeline Phases and Orchestration
The pipeline executes in six distinct phases to manage data dependencies and maximize throughput via concurrent fetch lanes.
The flow above illustrates the ordered execution of the pipeline, where Phase 1 establishes the universe before parallel fetching begins.
Sources: DO NOT DELETE EDL PIPELINE/README.md:139-147, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:257-380
Core Components and Roles
| Component | Responsibility | Relevant Files |
|---|---|---|
| Master Runner | Orchestrates script execution, lane concurrency, and error handling. | runner.py, run_full_pipeline.py |
| Universe Filter | Reconciles Dhan data with official NSE listings to enforce a Mainboard-only scope. | filter_mainboard_universe.py |
| Market Analyzer | Aggregates fundamental and technical data into the base JSON artifact. | bulk_market_analyzer.py |
| Breadth Engine | Calculates MBI/XP ratios, sector breadth, and index history. | mbi.py, process_mbi_market_breadth.py |
| Artifact Validator | Performs schema and count checks on generated files before publication. | artifacts.py, runner.py |
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:25-50, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:1-20, DO NOT DELETE EDL PIPELINE/filter_mainboard_universe.py:12-20
Frontend Architecture
The frontend is a static React application that functions as an interactive client for the pipeline's immutable data releases. It offloads heavy computation to Web Workers to maintain UI responsiveness.
Runtime Execution Flow
The application initializes by resolving the active release revision and loading the corresponding snapshot into a persistent Web Worker.
This diagram shows the asynchronous initialization and screening process within the browser.
Sources: frontend/README.md:195-210, frontend/README.md:323-335
Hybrid Condition Evaluation
The system utilizes a two-tier evaluation strategy. Standard filters and precomputed presets are processed in the browser. Complex historical queries or custom logic not present in the snapshot are proxied to a Python-based scanner bridge.
- Browser Engine: Handles Boolean expressions (
AND/OR), sorting, and pagination using published metrics. - Python Bridge: A Vite process feature in development that spawns a
python3worker to execute theTREND_CONDITION_ENGINE.
Sources: frontend/README.md:215-235, frontend/README.md:264-272
Data Models and Artifacts
Core Artifact: all_stocks_fundamental_analysis.json
This is the primary scanner artifact, standardized into "Scanner Schema v3". It contains identity, fundamental, technical, and event data for the entire mainboard universe.
| Field Category | Key Examples |
|---|---|
| Identity | Symbol, Name, Listing Date, Basic Industry, Sector |
| Fundamentals | Market Cap(Cr.), P/E, ROE(%), Sales Growth 5 Years(%) |
| Technical | RSI (14), SMA 200, RVOL, ADR(%), % from ATH |
| Events | Event Markers (ASM/GSM icons), Recent Announcements, News Feed |
Sources: DO NOT DELETE EDL PIPELINE/README.md:212-300, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:130-150
Market Breadth and MBI Ratios
The breadth engine generates specialized artifacts representing market health, including the "XP" (a publicly disclosed proxy for market strength).
# MBI Score Calculation Logic
green_count = sum(value == "green" for value in cells.values())
red_count = sum(value == "red" for value in cells.values())
score = green_count - red_count
state = "green" if score >= 3 else "red" if score <= -3 else "neutral"
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/mbi.py:115-121
System Integration and Deployment
The project is designed for static hosting with optional backend support for advanced features.
- Staging and Validation: The pipeline stages artifacts and validates them against
FINAL_ARTIFACT_SPECSbefore updating thecurrent.jsonpointer. - Compression: Artifacts are compressed to
.json.gzto reduce transport size (typically >70% reduction). - Static Hosting: The
frontend/distdirectory andpublic/datafiles are served via CDN/Static host. - R2 Charts: Candles and event records are published to Cloudflare R2 as content-addressed objects.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:136-150, frontend/README.md:290-310, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:195-220
The Nexus Scanner architecture ensures that research data is immutable and revision-pinned, allowing analysts to reproduce results from any specific trading session. All calculations, from technical indicators like SMA to advanced breadth metrics like XP, are derived solely from the validated data pipeline.
Sources: frontend/README.md:380-390, DO NOT DELETE EDL PIPELINE/README.md:320-325