Market Breadth & Indices Processing
Relevant source files
The following files were used as context for generating this wiki page:
- DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/pipeline.py
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py
- DO NOT DELETE EDL PIPELINE/README.md
- DO NOT DELETE EDL PIPELINE/tests/test_breadth_v2.py
- DO NOT DELETE EDL PIPELINE/build_rs_ratings.py
- DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py
Market Breadth & Indices Processing generates versioned market-breadth artifacts, index histories, and sector analytics for the Nexus Scanner. This system analyzes equity history and index data to calculate technical breadth indicators, including advances, declines, moving average relationships, and the proprietary MBI/XP (Market Breadth Index) metrics.
Sources: DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:1-12, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/pipeline.py:1-10
System Architecture and Data Flow
The breadth processing logic operates as part of Phase 4 (Injection) and Phase 2.5 (Indices) of the master pipeline. It consumes stock OHLCV data, index histories, and universe snapshots to produce compressed JSON artifacts used by the frontend.
Core Data Pipeline
The following diagram illustrates the flow from raw data sources to the final breadth artifacts.
The pipeline uses BreadthAccumulator objects to aggregate metrics across symbols and sectors simultaneously.
Sources: DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:73-95, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/pipeline.py:133-157
Breadth Generation Logic
The generate_market_breadth function coordinates the calculation of technical breadth across the eligible universe. It performs the following steps:
- Validates the methodology configuration.
- Builds a universe snapshot to identify eligible symbols based on price and market cap.
- Iterates through eligible symbols to load and prepare OHLCV history.
- Updates accumulators for the general market, specific indices (Nifty 50, Nifty 500, etc.), and sectors.
- Enriches aggregated records with MBI and XP ratios using index closes as a benchmark.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/pipeline.py:126-178, DO NOT DELETE EDL PIPELINE/tests/test_breadth_v2.py:207-228
Universe Eligibility
The system applies strict filters defined in breadth_methodology.json. Symbols must meet a minimum price and a market capitalization strictly greater than the threshold (e.g., ₹100 Crore).
Sources: DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:64-70, DO NOT DELETE EDL PIPELINE/tests/test_breadth_v2.py:118-125
Index History Processing
The index processing module standardizes history for all available indices. It identifies indices by their symbol and uses a unique IndexID to disambiguate collisions, such as multiple indices sharing the name "CAPINS".
Indices: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/indices.py, DO NOT DELETE EDL PIPELINE/tests/test_breadth_v2.py:349-366
Output Artifacts
The system produces several specific artifacts, which are then compressed into .json.gz format.
| Artifact File | Description |
|---|---|
market_breadth_v2.json | Main breadth records containing XP, MBI state, and MA percentages. |
all_indices_history_v2.json | Historical OHLCV and returns for all tracked indices. |
sector_breadth_v2.json | Breadth metrics broken down by industry sector. |
market_breadth_contributions_v2.json | Detailed lists of symbols contributing to specific metrics (audit log). |
breadth_universe_snapshot.json | The list of eligible and excluded symbols for the specific run. |
Sources: DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:22-31, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:61-69
Technical Schema
The market_breadth_v2.json artifact includes a table_schema for frontend rendering, mapping technical fields to human-readable labels:
- XP: Public-formula proxy for momentum/breadth strength.
- Ratio 4.5: Percentage of stocks meeting the specific "4.5" price/volume criteria.
- MA Percentages: Percentage of stocks above SMA/EMA 10, 20, 50, and 200.
Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/breadth/pipeline.py:16-41, DO NOT DELETE EDL PIPELINE/tests/test_breadth_v2.py:164-180
Quality Gates and Validation
The pipeline enforces strict history coverage requirements. The MINIMUM_HISTORY_COVERAGE constant is set to 0.90 (90%). If processed symbols or index history fall below this ratio relative to the eligible count, the script exits with an error to prevent publishing incomplete breadth data.
Sources: DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:32-35, DO NOT DELETE EDL PIPELINE/process_mbi_market_breadth.py:97-115
Summary
Market Breadth & Indices Processing provides the analytical foundation for the Nexus Scanner's market-wide views. By aggregating individual stock performance into sector and index-level metrics, it enables the detection of market-wide momentum shifts and technical breakouts. The system ensures data integrity through automated quality gates and uses atomic file operations to maintain consistent release artifacts.