Data Normalization Pipelines
Relevant source files
The following files were used as context for generating this wiki page:
Data Normalization Pipelines in the "WhereIsLifeCheaper" project serve as the critical bridge between raw, heterogeneous data extracted from various international supermarket and rental websites and a standardized, comparable database schema. The primary goal is to transform local variations in product names, pricing formats, units of measurement, and currencies into a unified format that enables cross-country cost-of-living analysis.
The system utilizes a multi-stage approach including text cleaning, quantity extraction, and currency conversion. This ensures that a product like "Coca-Cola® 1.5L PET" from Turkey is comparable to a similar entry in Uzbekistan or Spain through normalized identifiers and standardized unit pricing.
Sources: plan.md:12, plan.md:213-219, CLAUDE.md:12-16
Core Normalization Logic
The normalization process is triggered immediately after data extraction within individual scraper implementations and specialized utility services.
Product Name and Brand Normalization
Standardizing names involves removing special characters, converting to lowercase, and eliminating redundant whitespace. This process creates a normalized_name used for cross-country product matching.
Sources: plan.md:73-75, plan.md:213-221
Quantity and Unit Extraction
Scrapers use regex-based utilities to parse physical measurements (e.g., weight or volume) from raw strings. This logic handles various international formats such as "500g", "1.5L", "6 x 1.5 l", and "1Kg".
Sources: src/scrapers/germany/ReweScraper.ts:603-644, src/scrapers/albania/SparAlbaniaScraper.ts:258-274, src/scrapers/vietnam/WinMartScraper.ts:221-236
Data Flow Diagram
The following diagram illustrates the flow from raw scraper output to normalized database records.
The diagram shows the sequence of transformation steps applied to raw data before it enters the matching service and final storage.
Sources: plan.md:139-150, plan.md:154-162
Unit Normalization Standards
The system normalizes diverse measurement units into a standard set to facilitate "price per unit" calculations, which is essential for fair comparison.
| Raw Unit (Examples) | Standard Unit | Calculation Logic |
|---|---|---|
| kg, кг | kg | Identity |
| g, г | g / kg | Converted to kg if >= 1000g |
| l, liter, л | l | Identity |
| ml, мл | ml / l | Converted to l if >= 1000ml |
| stück, st., pcs, шт | pieces | Count-based normalization |
Sources: src/scrapers/germany/ReweScraper.ts:649-674, src/scrapers/uzbekistan/MakroScraper.ts:285-325, frontend/src/utils/currency.ts:62-81
Currency Normalization and Sync
To provide a consistent "Where is life cheaper" dashboard, the pipeline includes a synchronization service for exchange rates and a conversion utility.
Synchronization Pipeline
The sync-exchange-rates.ts script fetches daily rates from external APIs and updates the latest_exchange_rates table. It includes fallback logic to use hardcoded rates if the API is unavailable.
Sources: scripts/sync-exchange-rates.ts:22-54, scripts/sync-exchange-rates.ts:107-135
This diagram represents the automated daily workflow for maintaining accurate conversion data.
Sources: scripts/sync-exchange-rates.ts:56-140
Frontend Conversion Logic
The frontend utility convertToEUR applies these rates to localized prices for real-time comparison.
Sources: frontend/src/utils/currency.ts:55-58
Rental Data Normalization
The project extends normalization to real estate data, specifically focusing on bedrooms and standardized pricing (weekly vs. monthly).
Room to Bedroom Transformation
A specialized logic converts "rooms" (common in Eastern Europe) to "bedrooms" (common in international listings) by applying a rooms - 1 formula or detecting "studio" keywords.
Sources: src/scrapers/rent/tests/normalize.test.ts:13-24, src/scrapers/rent/tests/normalize.test.ts:106-118
Rental Feature Mapping
| Field | Normalization Action | Source Example |
|---|---|---|
| Bedrooms | "2-кімнатна" -> 1 Bedroom | normalize.test.ts:16 |
| Price | AUD Weekly -> AUD Monthly (x 4.33) | normalize.test.ts:121-140 |
| Area | "60 м²" -> 60 (numeric) | normalize.test.ts:114 |
Sources: src/scrapers/rent/tests/normalize.test.ts:106-140
Implementation Example: German Rewe Scraper
The ReweScraper demonstrates an advanced normalization implementation, handling multi-pack logic (e.g., "6 x 1,5 l") within the parsing phase.
// From src/scrapers/germany/ReweScraper.ts:607-616
const multiPackMatch = text.match(/(\d+)\s*x\s*(\d+[,.]?\d*)\s*(kg|g|l|ml|stück|st\.?|stk\.?)/i);
if (multiPackMatch) {
const count = parseInt(multiPackMatch[1], 10);
const unitSize = parseFloat(multiPackMatch[2].replace(',', '.'));
const totalQuantity = count * unitSize;
return this.normalizeUnit(unitType, totalQuantity);
}
Sources: src/scrapers/germany/ReweScraper.ts:607-616
Summary
The Data Normalization Pipelines are fundamental to the project's ability to match products across different languages and economic regions. By centralizing text cleaning, unit standardization, and currency synchronization, the system provides a reliable basis for the cross-country price comparisons displayed on the user dashboard.
Sources: plan.md:14-16, plan.md:154-156