Development Setup & Workflow
Relevant source files
The following files were used as context for generating this wiki page:
Development Setup & Workflow describes the environment configuration and operational procedures required to contribute to the Nexus Scanner project. This project consists of a script-first Python ETL pipeline (EDL Pipeline) and a React-based frontend application.
The workflow integrates data retrieval from public/undocumented NSE and Dhan endpoints with an interactive frontend research tool. Developers must follow specific setup steps for both the backend pipeline and the frontend runtime to ensure data consistency across the platform.
Sources: CONTRIBUTING.md:4-6, frontend/README.md:3-8, DO NOT DELETE EDL PIPELINE/README.md:377-380
Pipeline Environment Setup
The ETL pipeline requires a Python 3.9+ environment to execute data fetching, analysis, and enrichment scripts.
Backend Installation Steps
- Navigate to the pipeline directory:
cd "DO NOT DELETE EDL PIPELINE". - Create a virtual environment:
python3 -m venv .venv. - Activate the environment:
source .venv/bin/activate. - Install dependencies:
pip install -r requirements.txt. - (Optional) Perform an editable install for the
edl-pipelinecommand:pip install -e ..
Sources: CONTRIBUTING.md:8-13, DO NOT DELETE EDL PIPELINE/README.md:7-20
Configuration
The pipeline uses environment variables for runtime configuration. You can override default behaviors without modifying source code by setting these variables:
| Variable | Default | Description |
|---|---|---|
EDL_FETCH_OHLCV | True | Includes stock/index OHLCV sync. |
EDL_CLEANUP_INTERMEDIATE | True | Deletes intermediate JSON/CSV files after successful compression. |
EDL_BASE_DIR | Current Dir | Sets the absolute pipeline directory path. |
EDL_EOD2_DATA_DIR | N/A | Path to a local checkout of EOD2 data for historical bootstrapping. |
Sources: DO NOT DELETE EDL PIPELINE/README.md:27-39, DO NOT DELETE EDL PIPELINE/README.md:97-101
Frontend Development Setup
The frontend is built with React 19 and Vite 8, requiring Node.js 24 LTS.
Frontend Installation Steps
- Navigate to the frontend directory:
cd frontend. - Install Node dependencies:
npm ci. - Ensure the Python virtual environment from the pipeline is active to support the scanner bridge.
- Start the development server:
npm run dev -- --host localhost --port 8080.
Sources: frontend/README.md:46-55, frontend/README.md:74-88, frontend/package.json:10-28
Development Workflow Diagram
The following diagram illustrates the interaction between the local development environment and the data pipeline.
The diagram shows how the React UI consumes both immutable snapshots from the data pipeline and live evaluations from a Python bridge. Sources: frontend/README.md:93-100, frontend/README.md:321-326, frontend/publish_snapshot.py:100-110
Core Development Workflow
Local Verification
Before opening a pull request, you must run unit tests and compile checks to ensure script integrity:
- Run unit tests:
python3 -m unittest discover -s tests -v. - Check syntax:
python3 -m compileall -q ..
Sources: CONTRIBUTING.md:15-20, DO NOT DELETE EDL PIPELINE/README.md:395-399
Data Refresh Cycle
To generate fresh data for the frontend, execute the master runner. This process moves through distinct phases of data acquisition and transformation.
This sequence represents the standard data integration flow. Sources: DO NOT DELETE EDL PIPELINE/README.md:255-261, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:387-540
Contribution Rules
- Preserve Schemas: Maintain existing output field names unless performing an explicit migration.
- Atomic Writes: Ensure JSON and Gzip writes are atomic to prevent corrupted artifacts.
- Modular Logic: Shared logic (JSON parsing, HTTP requests) must reside in helper modules like
pipeline_utils.pyorohlcv_utils.py. - Do Not Commit Data: Generated intermediate files (raw JSON, CSV, news folders) must not be committed to Git.
Sources: CONTRIBUTING.md:28-40, DO NOT DELETE EDL PIPELINE/README.md:368-380
Data Publication Workflow
The application relies on immutable scanner revisions. The publication process freezes a dataset and provides it to the frontend.
Snapshot Generation
Running python3 frontend/publish_snapshot.py performs the following:
- Validates existing EDL artifacts.
- Computes browser-side metrics and evaluates preset outcomes.
- Freezes Python inputs into a content-addressed revision directory.
- Generates a new
current.jsonpointer for the frontend.
Sources: frontend/README.md:188-210, frontend/publish_snapshot.py:17-40
Artifact Specifications
The pipeline validates artifacts against specific criteria defined in artifacts.py.
| Script | Artifact | Type | Required Fields |
|---|---|---|---|
fetch_dhan_data.py | master_isin_map.json | JSON | Symbol, ISIN, Sid |
fetch_nse_corporate_actions.py | nse_corporate_actions.json | JSON | source, range, actions |
bulk_market_analyzer.py | all_stocks_fundamental_analysis.json | JSON | Symbol, Name, Sector, Market Cap(Cr.) |
standardize_stock_artifact.py | all_stocks_fundamental_analysis.json | JSON | REQUIRED_FINAL_FIELDS |
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:120-250
Troubleshooting & Maintenance
Common Setup Issues
- Port Conflicts: The frontend Vite server defaults to port 8080. Use
--strictPortto avoid automatic port hopping. - Bridge Timeouts: Local Python bridge requests have a 120-second timeout. Ensure the Python environment is active before starting Vite.
- Missing History: If local OHLCV caches are missing, the pipeline defaults to ~4 years of history for stock syncs.
Sources: frontend/README.md:93-100, frontend/README.md:321-326, DO NOT DELETE EDL PIPELINE/README.md:29-33
Manual History Recovery
If the Actions cache is evicted or local history is lost, use the recovery tool:
python3 sync_local_scanner_history.py \
--source-root /path/to/existing/EDL \
--from-date YYYY-MM-DD \
--as-of-date YYYY-MM-DD
Sources: frontend/README.md:386-395
The development workflow ensures that every release of the Nexus Scanner is backed by validated, reproducible data integrations. By combining strict backend verification with a component-driven frontend, the system maintains high data integrity for equity research.