Background Tasks & Scripts
Relevant source files
The following files were used as context for generating this wiki page:
The project utilizes a comprehensive suite of Bash and Node.js scripts to manage infrastructure automation, data synchronization, and repository health. These background tasks are primarily designed to run within CI/CD environments like GitHub Actions or via local administrative execution to ensure the integrity of the Cloudflare Pages deployment and the consistency of the D1 database.
Key functional areas covered by these scripts include synchronizing media assets from Backblaze B2 to Cloudflare D1, performing configuration snapshots of Cloudflare project metadata, and enforcing documentation standards through automated header validation.
Sources: scripts/B2_D1_SYNC_README.md:5-10, scripts/cf_pages_snapshot.sh:110-115, package.json:44-65
Media and Data Synchronization (B2 to D1)
The synchronization system is an end-to-end pipeline designed to migrate and index photo inventory from Backblaze B2 into the Cloudflare D1 photos table. This process is additive and idempotent, ensuring that only new objects are processed without duplicating existing records.
Pipeline Architecture
The synchronization follows a multi-step sequence:
- Inventory Generation: An AWS-compatible S3 API call fetches the current B2 object list.
- Metadata Enrichment: The inventory is enriched with additional metadata and public URL mappings.
- SQL Generation: A Node.js converter script transforms the enriched JSON into
INSERT OR IGNORESQL statements. - Execution: The generated SQL is executed against the remote D1 database using
wrangler.
Sources: scripts/b2_sync_photos_to_d1.sh:7-17, scripts/B2_D1_SYNC_README.md:65-75
The diagram above illustrates the data flow from external storage into the application's primary database.
Sources: scripts/b2_sync_photos_to_d1.sh:140-185, scripts/B2_D1_SYNC_README.md:65-75
Synchronization Configuration
| Variable | Source | Description |
|---|---|---|
B2_ENDPOINT | Env | S3-compatible endpoint URL for B2 |
B2_BUCKET | Env | Name of the source media bucket |
D1_DB_NAME | Env | Target D1 database (default: lgfc_lite) |
PUBLIC_B2_BASE_URL | Env | Public CDN URL for asset access |
DRY_RUN | Env | If set to 1, generates SQL without executing |
| Sources: scripts/b2_sync_photos_to_d1.sh:42-55, scripts/B2_D1_SYNC_README.md:40-55 |
Infrastructure Snapshots
To enable reproducible recovery and auditing, the cf_pages_snapshot.sh script captures the state of the Cloudflare Pages project. It fetches metadata, custom domain configurations, and recent deployment history, storing them as timestamped JSON files in the snapshots/cloudflare/ directory.
Captured Metadata
- Project Settings: Build commands, output directories, and environment variable names (values are redacted for security).
- Domains: Active custom domains and their DNS validation status.
- Deployments: The last three deployment records including commit SHAs and build status.
Sources: scripts/cf_pages_snapshot.sh:80-100, snapshots/cloudflare/README.md:15-35
The sequence above details the API interactions required to persist the current infrastructure state.
Sources: scripts/cf_pages_snapshot.sh:84-105
Repository Maintenance & Governance
Maintenance scripts handle repository metadata and enforce documentation standards within the CI pipeline.
Documentation Header Validation
The docs_check_headers.sh script ensures that all active markdown files contain a required YAML-like header. This header tracks document ownership, audience, and authority levels.
- Required Keys:
Doc Type,Audience,Authority Level,Owns,Does Not Own,Canonical Reference,Last Reviewed. - Target Paths:
docs/reference,docs/governance,docs/how-to,docs/explanation,docs/ops.
Sources: scripts/ci/docs_check_headers.sh:25-45, scripts/ci/docs_check_headers.sh:65-75
Repository Metadata Management
The update-repository-metadata.sh script automates the application of GitHub topics and repository descriptions using the GitHub CLI (gh). This ensures discoverability by tagging the repo with relevant keywords such as nextjs-15, tailwindcss, and cloudflare-pages.
Sources: scripts/update-repository-metadata.sh:44-70
Script Test Suites
The project includes integration tests specifically for background tasks to prevent regression in automation logic.
Incremental Sync Verification
The test_b2_d1_incremental_sync.sh script performs ten specific validations:
- Executability: Ensures the sync script is present and has +x permissions.
- Syntax: Validates Bash syntax using
bash -n. - Env Validation: Confirms the script fails gracefully when required variables like
CLOUDFLARE_API_TOKENare missing. - Idempotency: Verifies the use of
INSERT OR IGNOREin SQL generation. - Security: Checks for a
sql_escapefunction to prevent injection.
Sources: scripts/test_b2_d1_incremental_sync.sh:10-105
Summary of Commands
| Command | File Path | Purpose |
|---|---|---|
npm run snapshot | scripts/make-snapshot.mjs | Generic snapshot trigger |
npm run check:structure | scripts/check-repo-structure.mjs | Validates directory organization |
./scripts/cf_pages_snapshot.sh | scripts/cf_pages_snapshot.sh | Cloudflare infrastructure backup |
./scripts/b2_sync_photos_to_d1.sh | scripts/b2_sync_photos_to_d1.sh | Media inventory synchronization |
| Sources: package.json:57-65, scripts/cf_pages_snapshot.sh:5-10, scripts/b2_sync_photos_to_d1.sh:5-10 |