Cloudflare R2 Object Storage
Relevant source files
The following files were used as context for generating this wiki page:
Cloudflare R2 Object Storage serves as the primary remote storage backend for hosting immutable chart objects and scanner releases. The system stores compressed JSON data, including per-symbol candles, corporate actions, and regulatory announcements, enabling the frontend application to retrieve research data via content-addressed URLs.
The integration focuses on data immutability and archival. It preserves daily revisions for 90 days and moves the latest successful release of each month to a permanent archive. This ensures that the frontend can consistently resolve release pointers to specific, validated snapshots of market data.
Sources: frontend/README.md:329-335, frontend/chart_publication.py:9-11
System Architecture
The R2 storage system integrates with the Python EDL pipeline's publication phase. It utilizes a R2Store class to manage communication with Cloudflare using rclone as the underlying transfer tool. The architecture separates data into content-addressed "objects" and session-based "releases".
The diagram shows the flow of artifacts from the local pipeline to various R2 bucket partitions.
Sources: frontend/chart_publication.py:126-146, frontend/publish_snapshot.py:124-135
Storage Configuration
The system requires specific environment variables for Cloudflare R2 authentication and endpoint resolution. If these settings are missing, the pipeline defaults to local storage or issues warnings.
Required Environment Variables
| Variable | Description |
|---|---|
R2_ACCOUNT_ID | Cloudflare account identifier for S3 endpoint construction. |
R2_ACCESS_KEY_ID | S3-compatible access key for authentication. |
R2_SECRET_ACCESS_KEY | S3-compatible secret key for authentication. |
R2_PUBLIC_BASE_URL | The HTTPS delivery URL used by the frontend to fetch objects. |
R2_BUCKET | The target bucket name (defaults to nexus-screener-chart-data). |
Sources: frontend/chart_publication.py:23-28, frontend/chart_publication.py:130-132
Data Publication Logic
The publication process validates artifacts before transferring them to R2. The system supports two primary release schemas: Schema 6 (Direct Daily) and Schema 7 (Object-Based).
Object-Based Publication (Schema 7)
In this mode, individual chart files are treated as immutable objects identified by their SHA-256 hash. The R2Store.upload_objects method performs a differential sync by listing existing remote objects and only uploading new ones.
The sequence diagram illustrates the differential upload logic for content-addressed objects.
Sources: frontend/chart_publication.py:162-184, frontend/chart_publication.py:228-257
Monthly Archival
On the first release of a new month, the system automatically archives the previous month's latest successful release. This involves copying objects from the daily/ prefix to a monthly/ prefix to prevent them from expiring after the 90-day retention period.
Sources: frontend/chart_publication.py:192-200, frontend/chart_publication.py:218-223
Key Implementation Details
R2Store Class
The R2Store class encapsulates rclone commands. It sets specific metadata for uploads to ensure correct browser behavior, such as content-type=application/gzip and cache-control=public,max-age=31536000,immutable.
# frontend/chart_publication.py:149-156
def upload_charts(self, source, key):
# Application/gzip bytes are explicitly decompressed by the browser.
self.run('copy', str(source), self.remote(key), '--include', '*.json.gz',
'--transfers', '16', '--metadata', '--metadata-set', 'content-type=application/gzip',
'--metadata-set', 'cache-control=public,max-age=31536000,immutable', '--immutable', '--checksum')
self.run('check', str(source), self.remote(key), '--include', '*.json.gz', '--one-way', '--download')
Sources: frontend/chart_publication.py:148-158
Validation and Safety
Before any upload, the system performs a chart_preflight check. This ensures that:
- Chart artifact counts match the scanner release.
- The
asOfDatein chart indices aligns with the scanner session. - Content hashes match the actual file digests.
Sources: frontend/chart_publication.py:38-51, frontend/chart_publication.py:54-68
Conclusion
Cloudflare R2 Object Storage provides a scalable, immutable backend for Nexus Scanner. By leveraging content-addressing and automated archival, the system maintains high data integrity and availability for historical technical research while optimizing transfer costs through differential updates.
Sources: frontend/README.md:329-335, frontend/chart_publication.py:162-167