High-Level Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• CLAUDE.md
• plan.md
• README.md
• src/scrapers/malaysia/LotussApiScraper.ts
• src/api/routes/canonical.ts
• scripts/sync-exchange-rates.ts
• terraform/main.tf

The WhereIsLifeCheaper project is a multi-country grocery price comparison system designed to help users understand the cost of living across various regions. It functions by programmatically scraping supermarket websites daily, normalizing the data, and providing a web-based dashboard for price visualization and cross-country matching.

The architecture is built on a modular Node.js/TypeScript foundation, utilizing a PostgreSQL database for persistent storage and Playwright for robust web automation. The system is designed for high scalability, supporting ten countries including Turkey, Montenegro, Spain, Uzbekistan, Ukraine, Kazakhstan, Germany, Malaysia, Albania, and Austria.

Sources: CLAUDE.md:3-8, plan.md:3-13, README.md:1-12

Core System Components

The architecture is divided into five primary functional layers that handle data ingestion, processing, storage, and presentation.

1. Scraper Service Layer

This layer is responsible for the extraction of raw product data from various international supermarkets. It uses a modular architecture where a BaseScraper abstract class defines the standard lifecycle (initialize, scrape, cleanup), which is then extended by site-specific implementations such as MigrosScraper or LotussApiScraper.

2. Database Layer

A PostgreSQL 15 database serves as the central repository. The schema is optimized for time-series price data and cross-country product matching through the use of canonical product tables.

3. API Service

An Express.js REST API facilitates data access for the frontend and administrative tools. It handles authentication via Google OAuth 2.0 and provides endpoints for price history, country lists, and manual product mapping.

4. Scheduler and Utilities

Automation is handled via GitHub Actions or Node-cron, triggering daily syncs of exchange rates and supermarket scrapes. Utility modules manage logging (Winston), product name normalization, and retry logic with exponential backoff.

5. Frontend Dashboard

A React-based Single Page Application (SPA) built with Vite and TailwindCSS provides the user interface for price comparisons and administrative mapping tasks.

Sources: plan.md:15-30, CLAUDE.md:11-28, README.md:14-25

System Workflow and Data Flow

The following diagram illustrates the end-to-end data flow from the initial automated trigger to the user interface presentation.

Data Model and Schema

The database architecture is designed to handle the relationship between specific supermarket listings and a global "canonical" catalog that allows for cross-country comparisons.

Table Name	Primary Purpose	Key Fields
countries	Defines supported regions	code, name, currency_code
supermarkets	Registry of chains and scraper configs	name, base_url, scraper_class
products	Master catalog of unique items	name, normalized_name, brand, barcode
product_mappings	Links products to specific supermarkets	supermarket_id, external_id, url
prices	Time-series price records	price, currency, is_on_sale, scraped_at
canonical_products	Cross-country product identifiers	name, category_id, show_per_unit_price
exchange_rates	Currency conversion data	currency_code, rate_to_eur, source

Sources: plan.md:52-78, CLAUDE.md:37-46, src/api/routes/canonical.ts:101-135

Scraper Architecture

The scraper system uses a factory pattern to instantiate country-specific scrapers. Many modern implementations, like the LotussApiScraper, bypass standard HTML scraping in favor of intercepting internal API calls for increased efficiency and reliability.

Infrastructure and Deployment

The project is deployed using a hybrid approach, leveraging serverless functions for the API and containerized tasks for the scrapers.

• API Hosting: Vercel (Serverless).
• Database: PostgreSQL 15 (Docker for local, managed service for production).
• Scraper Execution: AWS ECS (Fargate) triggered by EventBridge or GitHub Actions.
• Infrastructure as Code: Terraform is used to manage AWS resources including ECR repositories, ECS clusters, and CloudWatch logs.

AWS Infrastructure Components (Terraform)

Resource	Description
aws_ecs_task_definition	Defines the container config for scrapers (2 vCPU, 4GB RAM)
aws_cloudwatch_event_rule	Schedule expression (cron: 0 3 * * ? *) for daily runs
aws_secretsmanager_secret	Secure storage for DATABASE_URL and PROXY_CONFIG
aws_ecr_repository	Private registry for the scraper Docker image

Sources: terraform/main.tf:31-105, 230-265, README.md:27-38, package.json:20-25

Technical Summary

The High-Level Architecture of WhereIsLifeCheaper emphasizes modularity and data integrity. By separating scraping logic into distinct classes and utilizing a robust canonical product mapping system, the project successfully bridges the gap between disparate international retail datasets. The use of Playwright with stealth plugins ensures resilience against bot detection, while the Terraform-managed AWS infrastructure provides a scalable environment for intensive daily data collection.

Sources: plan.md:267-275, CLAUDE.md:120-125