EDL Pipeline Runner
Relevant source files
The following files were used as context for generating this wiki page:
The EDL Pipeline Runner is the central orchestration system responsible for executing the End-of-Day (EOD) Data Layer (EDL) workflow. It manages the sequential and concurrent execution of scripts that fetch, analyze, and enrich Indian equity market data from sources like Dhan ScanX and the National Stock Exchange (NSE).
The runner ensures data integrity by validating outputs at each stage and producing a final, compressed, and validated dataset for consumption by the Nexus Scanner frontend. It supports phased execution—split into fetch and build phases—to allow for checkpoints and reliable publication via GitHub Actions.
Sources: [DO NOT DELETE EDL PIPELINE/README.md:14-25](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L14-L25), [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:1-8](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/runner.py#L1-L8)
Pipeline Architecture and Execution Phases
The pipeline operates in five primary phases to ensure that data dependencies are met. It uses a combination of serial execution for core data and parallel "lanes" for independent enrichment tasks.
Core Execution Flow
The diagram above illustrates the high-level dependency order of the pipeline stages.
Sources: [DO NOT DELETE EDL PIPELINE/README.md:188-196](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L188-L196), [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:279-370](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/runner.py#L279-L370)
Phase Summaries
| Phase | Description | Key Scripts |
|---|---|---|
| Phase 1 (Core) | Establishes the canonical universe. Downloads NSE listings and Dhan master maps. | fetch_dhan_data.py, filter_mainboard_universe.py |
| Phase 2 (Lanes) | Runs independent fetches in parallel (up to 3 workers). Covers filings, news, and technical indicators. | fetch_company_filings.py, fetch_all_ohlcv.py |
| Phase 3 (Analysis) | Builds the base JSON structure for all stocks. | bulk_market_analyzer.py |
| Phase 4 (Injection) | Injects advanced metrics (ADR, RVOL, RS Ratings) and corporate events into the master file. | advanced_metrics_processor.py, add_corporate_events.py |
| Phase 5 (Output) | Compresses artifacts to .json.gz and performs final validation checks. | standardize_stock_artifact.py |
Sources: [DO NOT DELETE EDL PIPELINE/README.md:188-196](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L188-L196), [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:46-93](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/artifacts.py#L46-L93)
Data Fetching and Concurrency
The runner manages network-intensive tasks using a ThreadPoolExecutor to execute independent script lanes. This avoids serial bottlenecks during slow fetches like company filings or OHLCV history.
Concurrent Lane Management
The run_script_lanes function limits top-level provider concurrency to a maximum of 3 workers. This ensures reasonable request volume to upstream endpoints while maintaining throughput.
# From src/edl_pipeline/runner.py
with ThreadPoolExecutor(max_workers=min(3, len(lanes)), thread_name_prefix="edl-fetch") as executor:
futures = {
name: executor.submit(run_script_sequence, scripts)
for name, scripts in lanes.items()
}
Sources: [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:133-142](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/runner.py#L133-L142), [DO NOT DELETE EDL PIPELINE/README.md:92-95](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L92-L95)
OHLCV Fetch Lane
The OHLCV lane follows a strict internal order to ensure price history is correctly merged and adjusted:
fetch_nse_delivery_history.py: Retrieves delivery data for session verification.import_eod2_ohlcv.py: Seeds adjusted history from local EOD2 repositories.apply_nse_daily_ohlcv.py: Applies the latest official NSE bhavcopy.fetch_all_ohlcv.py: Performs incremental backfill from Dhan.
Sources: [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:65-73](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/artifacts.py#L65-L73)
Validation and Integrity Framework
Integrity is maintained through a "Fail-Fast" mechanism. Every script execution is followed by an artifact validation step.
Validation Levels
- Per-Script Validation: After a script runs, the runner checks for the existence and basic schema of its specific outputs (e.g.,
master_isin_map.jsonmust contain "Symbol", "ISIN", and "Sid"). - Critical Failures: Scripts marked as
required=True(like Phase 1 scripts) will stop the entire pipeline if they fail or if their output validation fails. - Final Artifact Validation: Before publication, the runner validates the final
.json.gzfiles against theREQUIRED_FINAL_FIELDSschema.
Sources: [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:65-115](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/runner.py#L65-L115), [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:114-165](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/artifacts.py#L114-L165)
Pipeline Reporting
The runner generates a pipeline_report.json file. This report contains:
- Execution status and return codes for every script.
- Elapsed time for each stage.
- Validation results and specific error messages.
- Final byte sizes (raw vs compressed) of artifacts.
- Configuration flags used during the run.
Sources: [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:246-258](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/runner.py#L246-L258), [DO NOT DELETE EDL PIPELINE/README.md:27-30](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L27-L30)
Publication and Promotion
The pipeline uses an atomic publication strategy to ensure that consumers never see a partially updated or invalid dataset.
Staging and Promotion Flow
- Staging: The pipeline builds all artifacts in a temporary directory (stage).
- Quality Inspection: The
inspect_publicationfunction verifies the complete output set for freshness and consistency. - Atomic Promotion: If all checks pass, the runner replaces the published files using a filesystem rename. If an I/O failure occurs during promotion, the system rolls back to the previous backup.
Sources: [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py:34-75](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/publication.py#L34-L75), [DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py:149-165](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/src/edl_pipeline/publication.py#L149-L165)
Configuration and Environment
The runner behavior is controlled via environment variables, which are parsed by the PipelineConfig class.
| Variable | Description | Default |
|---|---|---|
EDL_FETCH_OHLCV | Includes stock/index OHLCV synchronization. | True |
EDL_FETCH_OPTIONAL | Includes optional ETF and F&O standalone data. | False |
EDL_CLEANUP_INTERMEDIATE | Deletes JSON/CSV files after successful compression. | True |
EDL_BASE_DIR | Sets the root directory for all pipeline operations. | Current Dir |
Sources: [DO NOT DELETE EDL PIPELINE/README.md:32-40](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/README.md#L32-L40), [DO NOT DELETE EDL PIPELINE/pipeline_utils.py:44-46](https://github.com/Anny26022/nexus-scanner/blob/65377ef377f5dc198ffb96d14d7f0d4fcf49cc00/DO%20NOT%20DELETE%20EDL%20PIPELINE/pipeline_utils.py#L44-L46)
Summary
The EDL Pipeline Runner provides a robust, fail-safe environment for processing financial data. By enforcing strict script sequencing, implementing parallel fetch lanes, and requiring multi-level artifact validation, it ensures that the Nexus Scanner operates on accurate and complete market snapshots. The system's ability to run in phases and roll back on failure makes it suitable for automated daily refreshes in production environments.