Project Overview

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• CLAUDE.md
• plan.md
• src/api/server.ts
• src/database/migrations/005_create_product_mappings_and_prices.sql
• package.json

WhereIsLifeCheaper is a multi-country grocery price comparison system designed to help users compare the cost of living across diverse regions, including Turkey, Montenegro, Spain, Uzbekistan, Ukraine, Kazakhstan, Germany, Malaysia, Albania, and Austria. The system automates the daily collection of price data from international supermarket chains, tracks historical price fluctuations, and provides a web-based interface for data visualization and cross-country matching.

Sources: README.md:3-8, CLAUDE.md:3-7

The project architecture comprises a Node.js/TypeScript backend utilizing Playwright for browser automation, a PostgreSQL database for time-series price storage, and a React-based frontend. It features a modular scraper framework that enables the system to scale across different supermarket web architectures, including those using REST APIs and those requiring complex headless browser interactions to bypass bot detection.

Sources: CLAUDE.md:10-18, plan.md:12-21

System Architecture

The application follows a modular architecture consisting of several integrated services. The Backend is built on Express.js and serves as the orchestration layer for scraping jobs and the REST API. The Database layer manages the persistence of geographical data, product catalogs, and historical price points.

Sources: CLAUDE.md:20-37, plan.md:24-63

Architecture Diagram

The following diagram illustrates the high-level flow of data from external supermarket websites through the scraper service into the database, and finally to the user via the API and Frontend.

Core Components

1. Scraper Service

The scraper service is the engine of the project, responsible for automated data collection. It utilizes an abstract BaseScraper class that provides common functionality like Playwright initialization, retry logic, and error handling. Specific implementations (e.g., MigrosScraper, ReweScraper) extend this base to handle site-specific selectors and navigation.

Feature	Description
Playwright Integration	Uses Chromium for browser automation and playwright-extra with stealth plugins to bypass Cloudflare.
Modular Design	ScraperFactory instantiates the correct scraper based on database-driven configuration.
Error Resilience	Configurable retries (default 3) and screenshot capture on failure.
API Scraping	Optimized scrapers (like Lotus's or Migros) use direct API calls where possible for efficiency.

Sources: CLAUDE.md:10-18, plan.md:120-134, src/scrapers/malaysia/LotussApiScraper.ts:145-160

2. Database Schema

The system uses a PostgreSQL 15+ database to store relational data and historical price points. The schema is designed to support cross-country product matching through a canonical product system.

Table	Purpose
countries	Stores country codes (TR, ME, ES, etc.) and currency information.
supermarkets	Links to countries and stores scraper configurations.
products	Master catalog of normalized product names and brands.
product_mappings	Links products to specific supermarket URLs and external IDs.
prices	Historical time-series data including sale status and price per unit.
canonical_products	Identifiers for matching equivalent products across different countries.

Sources: CLAUDE.md:40-47, plan.md:83-110, src/database/migrations/005_create_product_mappings_and_prices.sql:1-60

3. API Service

The Express.js server provides several endpoints for both public consumption and administrative management. It includes health monitoring and SEO support via dynamic sitemaps.

Method	Endpoint	Description
GET	/api/countries	Retrieves list of supported countries.
GET	/api/products	Searches the product catalog with filters.
GET	/api/prices/latest	Fetches most recent price data for products.
GET	/api/canonical	Lists cross-country matched products.
POST	/api/scraper/trigger	Manually starts a scraping job (Admin only).
GET	/health	Returns system and database connectivity status.

Sources: README.md:113-138, src/api/server.ts:86-130

Data Workflow

The system operates on a scheduled cycle to maintain up-to-date information.

Technology Stack

The project utilizes a modern TypeScript stack across the entire pipeline.

{
  "dependencies": {
    "express": "^4.18.2",
    "playwright": "^1.40.0",
    "pg": "^8.11.3",
    "winston": "^3.11.0",
    "passport-google-oauth20": "^2.0.0",
    "react": "^19.0.0"
  }
}

Sources: package.json:44-70, README.md:14-23

Key Implementation Logic

Product Normalization and Matching

A critical feature of the system is the ability to match products across different markets. This is achieved through a multi-step process:

1. Normalization: Removing special characters and standardizing whitespace.
2. Quantity Extraction: Parsing units like "1.5L" or "500g" into numeric values and standard unit strings.
3. Matching Priority:
   • Exact barcode match.
   • Brand + normalized name + unit match.
   • Fuzzy matching within categories.
   • Manual mapping via the Admin UI.

Sources: CLAUDE.md:126-131, plan.md:142-151

Exchange Rate Synchronization

To allow fair comparison, the system includes a synchronization script that fetches rates relative to the Euro (EUR) from the currency-api. This enables the frontend to display prices in a common currency regardless of the local supermarket's currency (e.g., TRY, ALL, MYR).
Sources: scripts/sync-exchange-rates.ts:11-40

Summary

WhereIsLifeCheaper provides a robust framework for international cost-of-living analysis. By combining automated scraping, strict data normalization, and a centralized database, it enables developers and users to track grocery market trends across ten countries. The system's extensibility via its modular scraper architecture allows for the continuous addition of new supermarkets and regions.

Sources: README.md:218-223, plan.md:278-285