Chart Artifact Generation

Relevant source files

The following files were used as context for generating this wiki page:

• DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py
• frontend/chart_publication.py
• DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py
• DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py
• DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py
• frontend/README.md

Chart Artifact Generation produces compact, immutable, per-symbol JSON payloads containing OHLCV data, corporate actions, and news events. These artifacts allow the Nexus Scanner frontend to display detailed technical charts and historical event markers without downloading the entire scanner dataset. The system clips all generated data to the same completed NSE trading session as the primary scanner release to ensure data consistency.

Sources: DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:1-10, frontend/README.md:275-290

Architecture and Data Flow

The generation process occurs in Phase 4.5 of the EDL Pipeline, after the canonical stock artifact is standardized but before intermediate files are cleaned up. The builder aggregates data from multiple source files, including OHLCV CSVs, corporate action ledgers, and news feed directories.

The diagram shows how raw pipeline data is transformed into compressed symbol objects and indexed for either local or R2-based publication.

Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/artifacts.py:100-110, DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:148-180

Component Specifications

1. Data Aggregation and Processing

The build_chart_artifacts.py script functions as the primary builder. It iterates through the symbols defined in the canonical stock artifact and performs the following actions:

• OHLCV Processing: Loads daily candles from ohlcv_data/, filtering by the current session date.
• Volume Event Detection: Identifies monthly, quarterly, and yearly volume extrema (highest/lowest ever) for technical analysis.
• Event Injection: Merges classified corporate actions from the corporate_action_ledger.json and recent news items from the market_news/ directory.

Sources: DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:65-90, DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:105-120

2. Payload Structure

Each symbol object is stored as a compressed JSON file (.json.gz). The payload includes the following key structures:

Field	Description	Retention/Limit
historyStartDate	ISO date of the first available candle.	All available history
candles	Array of OHLCV objects.	Up to current session
volumeEvents	Highest/lowest volume markers for different periods.	60M, 20Q, 10Y
corporateActions	Ex-date, subject, and adjustment factors.	Verified actions only
marketNews	Top 50 real-time news items with sentiment analysis.	Latest 50 items

Sources: DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:170-185, frontend/README.md:465-480

3. Artifact Indexing and Hashing

The system generates an index.json file to map symbols to their respective object hashes. This ensures immutability and facilitates content-addressed storage.

The diagram illustrates the relationship between the central Index and individual per-symbol Chart Objects.

Sources: DO NOT DELETE EDL PIPELINE/build_chart_artifacts.py:182-187, frontend/chart_publication.py:45-60

Publication and Storage

The system supports two primary storage backends: local filesystem and Cloudflare R2. The chart_publication.py module manages the promotion of artifacts from the staging directory to the production environment.

R2 Publication Workflow

When R2 storage is enabled via EDL_CHART_STORAGE=r2, the system follows a specific sequence:

1. Object Upload: Uploads new .json.gz objects to the objects/ prefix using rclone.
2. Metadata Setting: Sets Content-Type: application/gzip and Cache-Control: public,max-age=31536000,immutable.
3. Integrity Check: Performs a one-way checksum verification after upload.
4. Index Promotion: Writes the release.json manifest only after all objects are verified.

Sources: frontend/chart_publication.py:146-170, frontend/chart_publication.py:270-290

Local Promotion and Safety

For local deployments, the system uses an atomic promotion strategy to prevent concurrent readers from accessing partially written data.

• Atomic Copy: Uses same-filesystem renames (replace) for individual files.
• Directory Swapping: Moves existing directories to a .previous backup before moving new artifacts into place.
• Rollback: If an exception occurs during promotion, the system restores files from the backup directory.

Sources: DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py:25-50, frontend/chart_publication.py:290-305

Validation Gates

Before a chart release is promoted, it must pass multiple integrity checks:

• Session Alignment: The asOfDate in the chart index must match the sessionDate of the scanner release.
• Hash Verification: Every object filename must match the SHA256 digest of its content.
• Universe Consistency: The number of symbols in the index must match the number of generated objects.
• Schema Version: The system validates that objects conform to schemaVersion: 2.

Sources: frontend/chart_publication.py:45-70, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:200-215

Summary

Chart Artifact Generation ensures that the Nexus Scanner provides high-performance charting capabilities while maintaining strict data integrity. By decoupling chart candles and events from the main scanner JSON, the project minimizes initial load times and enables content-addressed caching for historical data. The use of atomic promotion and checksum-verified R2 uploads protects the production environment from partial or corrupted data refreshes.

Sources: frontend/README.md:435-450, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/publication.py:90-100