Background Jobs & CLI Scripts
Relevant source files
The following files were used as context for generating this wiki page:
The "WhereIsLifeCheaper" project relies on a robust suite of background jobs and Command Line Interface (CLI) scripts to automate data collection, maintain data integrity, and perform system administrative tasks. These utilities handle the daily orchestration of web scrapers across multiple countries, synchronize financial exchange rates, and provide tools for developers to test and refine scraper logic without deploying to production environments.
Most background tasks are scheduled to run daily via automated triggers. In the development environment, these are executed using ts-node, while production deployments leverage AWS EventBridge to trigger ECS Fargate tasks or GitHub Actions for scheduled execution. Sources: plan.md:141-143, terraform/main.tf:316-322, package.json:11-20.
Data Collection and Scraper Orchestration
The core background activity is the automated scraping of supermarket websites. This is handled by a dedicated service that can run scrapers in parallel or individually.
Scraper Execution Script (run-scraper.ts)
This script acts as the primary entry point for running scrapers in a production or staging context. It utilizes the ScraperService to manage execution, handle concurrency, and log results to the PostgreSQL database.
The script supports several flags for granular control:
--concurrency=N: Sets the number of scrapers to run in parallel (default: 3).--categories=id1,id2: Filters execution to specific categories.--list-categories: Displays available categories for a given supermarket.
Sources: scripts/run-scraper.ts:15-22, scripts/run-scraper.ts:31-34.
Scraper Testing Script (test-scrapers.ts)
For development purposes, test-scrapers.ts allows manual validation of scraper logic without persisting data to the primary database tables. It bypasses the ScraperService and interacts directly with the BaseScraper implementations and the scraperRegistry.
Sources: scripts/test-scrapers.ts:1-20.
Financial Data Synchronization
To facilitate cross-country price comparisons, the system must maintain up-to-date exchange rates.
Exchange Rate Sync (sync-exchange-rates.ts)
This job synchronizes currency rates relative to the Euro (EUR). It follows a multi-tiered fetch strategy to ensure reliability.
| Feature | Description |
|---|---|
| Primary Source | fawazahmed0/currency-api via JSDelivr CDN. |
| Fallback Source | Hardcoded FALLBACK_EXCHANGE_RATES and Cloudflare mirrors. |
| Integrity Check | Detects large rate changes (>10%) and logs warnings. |
| Data Storage | Rates are saved to the exchange_rates table with source metadata. |
Sources: scripts/sync-exchange-rates.ts:14-20, scripts/sync-exchange-rates.ts:89-106.
Infrastructure & Scheduling
The project defines its scheduled jobs via infrastructure-as-code and environment-specific triggers.
Scheduled Job Configurations
| Job Name | Schedule | Environment | Trigger Mechanism |
|---|---|---|---|
| Daily Scrape | 03:00 AM UTC | Production | AWS EventBridge / ECS Fargate |
| Scraper Run | Daily | CI/CD | GitHub Actions (cron: '0 3 * * *') |
| Exchange Sync | On Demand/Daily | All | npm run rates:sync |
Sources: terraform/main.tf:316-322, README.md:167-172, package.json:18.
CLI Command Summary
The package.json file defines standard commands for interacting with these scripts:
# Data Scraping
npm run scraper:run # Execute production scrapers
npm run scraper:test # Test scraper output in console
# Maintenance
npm run migrate # Run database migrations
npm run seed # Load initial countries/supermarkets
npm run rates:sync # Update exchange rates
npm run build # Compile TypeScript to dist/
Sources: package.json:10-25.
Specialized Maintenance Scripts
Occasionally, scripts are created to fix data inconsistencies resulting from changing scraper logic.
REWE Unit Update (update-rewe-product-units.ts)
This script targets a specific data issue where REWE products were incorrectly assigned "pieces" as a unit despite having weight information (e.g., "500g") in their names. It uses regex patterns like /\b(\d+[,.]?\d*)\s*(kg|g|ml|l)\b/i to retroactively parse and update the products table.
Sources: scripts/update-rewe-product-units.ts:1-25.
Conclusion
Background jobs and CLI scripts form the operational backbone of the "WhereIsLifeCheaper" platform. By separating development testing (test-scrapers.ts) from production orchestration (run-scraper.ts) and automating environmental data synchronization (sync-exchange-rates.ts), the architecture ensures the system remains accurate and scalable with minimal manual intervention.