Snapshot Publication & Immutability

Relevant source files

The following files were used as context for generating this wiki page:

• frontend/publish_snapshot.py
• frontend/packed_snapshot.py
• frontend/src/api/realAdapter.ts
• frontend/README.md
• DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py
• frontend/test_publish_snapshot.py

Nexus Scanner uses an immutable data release system to combine equity research with validated data snapshots. The Python EDL (Extraction, Data, Loading) pipeline generates these snapshots, which the frontend consumes as content-addressed revisions. This architecture ensures that specific scanner results are reproducible by pinning them to a unique revision hash.

The system facilitates research by identifying the exact session and revision behind any specific result. It enforces data integrity by rejecting backend responses that do not match the immutable revision currently active in the application.

Sources: frontend/README.md:1-12, frontend/README.md:329-335

Publication Architecture

The publication process transforms validated EDL artifacts into a content-addressed scanner revision. The publisher reads source data, computes browser-ready metrics, and freezes inputs to ensure historical consistency.

Data Flow and Hashing

The publication engine calculates a SHA256 digest based on the source data, delivery records, chart artifacts, and the state of the scanner's implementation code. This digest becomes the revision ID for the release.

The diagram shows how source artifacts and code contribute to the final content-addressed revision hash.
Sources: frontend/publish_snapshot.py:84-118, frontend/README.md:298-315

Artifact Generation

When a new snapshot is published, the system creates a specific directory structure under public/data/revisions/<revision>/.

Artifact	Format	Purpose
stocks.json	JSON	Uncompressed equity snapshot for compatibility.
stocks.json.gz	Gzip JSON	Compressed snapshot for supported browsers.
stocks.packed.json.gz	Packed Gzip	Columnar-encoded snapshot for optimized worker loading.
ipos.json.gz	Gzip JSON	IPO catalogue specific to the release session.
release.json	JSON	Immutable release metadata including schema version.
financial-history.json.gz	Gzip JSON	Full historical financial series linked by symbol.

Sources: frontend/publish_snapshot.py:133-176, frontend/README.md:310-322

Snapshot Packaging & Optimization

The system employs a "packed" format to reduce the memory footprint and transfer size of equity data. This format uses columnar encoding rather than an array of objects.

Packed Snapshot Structure

The pack_snapshot function identifies common keys across all stock records and stores the values in a 2D array. It tracks missing fields using a bitmask or coordinate list to ensure data integrity during reconstruction.

def pack_snapshot(payload):
    stocks = payload['stocks']
    if not stocks:
        return {**payload, 'stockEncoding': 'columnar-v1'}
    keys = sorted(stocks[0].keys())
    values = []
    missing = []
    for i, stock in enumerate(stocks):
        row = []
        for key in keys:
            if key in stock:
                row.append(stock[key])
            else:
                row.append(None)
        values.append(row)
    return {
        **payload,
        'stocks': [],
        'stockEncoding': 'columnar-v1',
        'stocksTable': {'keys': keys, 'values': values, 'missing': missing}
    }

Sources: frontend/packed_snapshot.py:4-44

Immutability and Frontend Integration

The frontend application maintains synchronization with the published data through a manifest system.

The Manifest Lifecycle

1. Discovery: The app fetches /data/current.json to identify the active revision.
2. Validation: The realAdapter.ts validates the manifest schema, revision format, and URL paths.
3. Pinning: All subsequent requests for stocks, charts, or IPO data include the revision ID to ensure consistency.

The sequence diagram illustrates the frontend fetching the pointer file before requesting revision-pinned artifacts.
Sources: frontend/src/api/realAdapter.ts:119-155, frontend/README.md:329-335

Manifest Fields

The Manifest interface defines the contract for a published release.

Field	Type	Description
revision	string	64-character SHA256 hex digest.
sessionDate	string	ISO date of the trading session.
datasetGzipUrl	string	Path to the compressed stock snapshot.
datasetPackedGzipUrl	string	Path to the columnar-encoded snapshot.
financialHistoryUrl	string	Link to the immutable financial history archive.

Sources: frontend/src/api/realAdapter.ts:7-21

Publication Safety and Verification

The run_full_pipeline.py and publish_snapshot.py scripts implement gates to prevent corrupted data from reaching users.

1. Atomic Writes: The publisher writes artifacts to .tmp files and replaces existing files only after successful write completion.
2. Revision Locking: The system raises a RuntimeError if the source EDL data changes while a snapshot is being published.
3. Artifact Validation: The pipeline runner validates final artifacts (byte size, schema, count) before updating the current.json pointer.
4. Frozen Inputs: Frozen Python inputs are stored in a private directory linked to the revision hash to ensure server-side evaluation matches the client-side snapshot.

Sources: frontend/publish_snapshot.py:15-22, frontend/publish_snapshot.py:115-118, DO NOT DELETE EDL PIPELINE/src/edl_pipeline/runner.py:207-220

Snapshot Publication ensures that every equity research session is anchored to a specific, immutable point in time. By combining content-addressable storage with columnar data optimization, the system provides both high performance and total reproducibility for technical and fundamental screening.