Backblaze B2 Media Storage
Relevant source files
The following files were used as context for generating this wiki page:
Backblaze B2 serves as the primary external media storage for the Lou Gehrig Fan Club (LGFC) project, housing images and assets that are too large for direct inclusion in the repository. The system is designed to provide highly available media assets while maintaining a reference index within a Cloudflare D1 database to support site features like the Photo Archive and Memorabilia galleries.
Sources: context.md:14, scripts/B2_D1_SYNC_README.md:3-5
Architecture and Integration
The B2 storage system integrates with the application through two primary paths: a runtime library used by Cloudflare Workers and a set of synchronization scripts that maintain parity between B2 object storage and Cloudflare D1 database tables.
System Overview
The diagram above illustrates how media is stored in B2, indexed in D1, and served via Cloudflare Workers or direct CDN links.
Sources: functions/_lib/b2.ts:1-10, scripts/b2_d1_incremental_sync.sh:5-15
Core Components
| Component | Responsibility | Source Files |
|---|---|---|
| B2 Library | Provides S3-compatible XML parsing and object listing for Workers. | functions/_lib/b2.ts |
| Incremental Sync | Idempotent bash script to detect and sync new B2 objects to D1. | scripts/b2_d1_incremental_sync.sh |
| Media Ingest | Node.js utility for generating stable UIDs and performing SQL ingestion. | scripts/d1_media_ingest.js |
| Daily Sync | Automated workflow for delta-based synchronization. | scripts/b2_d1_daily_sync.sh |
Sources: functions/_lib/b2.ts, scripts/b2_d1_incremental_sync.sh, scripts/d1_media_ingest.js, scripts/b2_d1_daily_sync.sh
Synchronization Pipeline
The project employs a delta-based synchronization strategy to ensure that new media uploaded to B2 is indexed in the photos table of the D1 database without creating duplicates.
Incremental Sync Flow
The synchronization process follows a strict sequence to maintain idempotency:
The diagram shows the logic used to prevent redundant data processing by comparing B2 keys against D1 before execution.
Sources: scripts/b2_d1_incremental_sync.sh:135-260, scripts/B2_D1_SYNC_README.md:73-82
Data Mapping
When B2 objects are ingested into the D1 photos table, specific metadata is mapped to database columns:
| D1 Column | B2 Source | Logic / Transformation |
|---|---|---|
photo_id | Object Key | Used as the stable external identifier. |
url | Key + Base URL | Prefixed with PUBLIC_B2_BASE_URL. |
created_at | LastModified | ISO8601 timestamp from B2. |
media_uid | fileId + Key | SHA-256 hash (first 40 chars) prefixed with b2_. |
Sources: scripts/B2_D1_SYNC_README.md:65-71, functions/_lib/b2.ts:65-71, scripts/d1_media_ingest.js:52-59
Technical Implementation Details
Configuration and Authentication
Access to B2 requires specific environment variables that enable S3-compatible authentication. The requireB2 function validates these at runtime.
// functions/_lib/b2.ts:15-20
export type B2Bindings = {
B2_ENDPOINT: string;
B2_BUCKET: string;
B2_KEY_ID: string;
B2_APP_KEY: string;
};
Sources: functions/_lib/b2.ts:15-30, scripts/B2_D1_SYNC_README.md:39-46
Stable Identifier Generation
To ensure media assets can be tracked reliably even if file names change (where fileId is available), the system generates a media_uid.
// scripts/d1_media_ingest.js:52-59
function generateMediaUid(b2FileId, b2Key) {
const combined = `${b2FileId}:${b2Key}`;
const hash = createHash('sha256').update(combined).digest('hex');
return `b2_${hash.substring(0, 40)}`;
}
Sources: scripts/d1_media_ingest.js:52-59, functions/_lib/b2.ts:65-71
XML Parsing for S3 Compatibility
Since Backblaze B2 provides an S3-compatible API, the project includes custom XML parsing logic to handle ListObjectsV2 responses within the Cloudflare environment where full S3 SDKs might be too heavy.
// functions/_lib/b2.ts:46-59
export function parseListObjectsV2Xml(xml: string): ListedB2Object[] {
const out: ListedB2Object[] = [];
const parts = xml.split("<Contents>");
for (let i = 1; i < parts.length; i++) {
const block = parts[i].split("</Contents>")[0];
const key = extractXmlTag(block, "Key");
// ... metadata extraction ...
out.push({ key, size, etag, fileId, lastModified });
}
return out;
}
Sources: functions/_lib/b2.ts:46-59
Maintenance and Operations
Error Handling
The synchronization scripts use explicit exit codes to facilitate automated monitoring in CI/CD pipelines:
- Code 1: Missing environment variables.
- Code 2: B2 connection/authentication failure.
- Code 3: D1 query failure.
- Code 4: SQL execution failure.
Sources: scripts/B2_D1_SYNC_README.md:55-63, scripts/b2_d1_incremental_sync.sh:25-35
Idempotency Guarantees
The system maintains idempotency through two layers:
- Delta Calculation: The scripts compare the B2 object list against existing keys in D1 before generating SQL.
- SQL Constraints: Use of
INSERT OR IGNOREorWHERE NOT EXISTSclauses ensures that even if the same script runs twice, no duplicate records are created.
Sources: scripts/b2_d1_incremental_sync.sh:225-245, scripts/B2_D1_SYNC_README.md:84-88
Backblaze B2 Media Storage provides a robust, externalized asset management system for the LGFC project. By utilizing S3-compatible APIs and automated synchronization scripts, the project ensures that high-resolution media is efficiently indexed and accessible via Cloudflare's edge network.