Scraper Backend Service
Relevant source files
The following files were used as context for generating this wiki page:
The Scraper Backend Service is a modular, automated system designed to extract grocery pricing data from various international supermarket chains. It serves as the data acquisition layer for the "WhereIsLifeCheaper" platform, enabling cross-country cost-of-living comparisons by daily tracking of product names, prices, units, and availability.
Sources: plan.md:3-8, CLAUDE.md:3-6
The system is built on a modular architecture using Node.js and TypeScript, utilizing Playwright for browser-based automation and direct API/GraphQL integrations for high-performance extraction where available. It supports a wide array of countries including Turkey, Montenegro, Spain, Germany, Ukraine, and others.
Sources: CLAUDE.md:14-25, plan.md:175-188
Core Architecture
The service follows a template method pattern centered around an abstract base class. This ensures consistent error handling, logging, and browser management across all supermarket implementations.
Base Scraper Logic
All individual scrapers extend the BaseScraper class. This class provides standardized methods for browser initialization, navigation with retry logic, anti-bot mitigation, and result building. It manages the lifecycle of a scraping session, including starting a runId and tracking statistics like productsScraped and productsFailed.
Sources: src/scrapers/base/BaseScraper.ts:16-55, CLAUDE.md:46-51
The diagram above illustrates the inheritance hierarchy where concrete scrapers implement specific extraction logic while inheriting core functionality.
Extraction Strategies
The service employs three distinct strategies based on the target website's architecture:
- Browser Automation (Playwright): Used for JavaScript-heavy sites requiring user interaction (e.g., selecting delivery zones or solving challenges).
- REST API Interception: Captures JSON responses from internal web APIs to increase speed and data accuracy (e.g., WinMart, Lotus's).
- Direct GraphQL/API Calls: High-performance extraction that bypasses browser overhead entirely by communicating directly with backend endpoints (e.g., Auchan Ukraine).
Sources: src/scrapers/ukraine/AuchanUaGraphQLScraper.ts:121-125, src/scrapers/germany/ReweScraper.ts:98-105, plan.md:180-185
Scraping Workflow and Scheduling
The scraping process is orchestrated through a service layer that manages concurrency and logging.
Execution Flow
A typical scraping run involves initializing the specific scraper, navigating through category configurations, paginating through product lists, and saving results incrementally.
This flow represents the standard lifecycle of a scraping task as defined in the BaseScraper's template methods.
Sources: src/scrapers/base/BaseScraper.ts:77-111, scripts/run-scraper.ts:31-45
Automation and Reliability
- Daily Scheduling: Automated via GitHub Actions or EventBridge, typically running at 03:00 UTC.
- Retry Logic: The system uses exponential backoff for network failures, retrying operations up to 3 times by default.
- Anti-Bot Handling: Includes random mouse movements, scrolling, user-agent rotation, and the
playwright-extrastealth plugin to bypass Cloudflare and Turnstile challenges.
Sources: src/scrapers/base/BaseScraper.ts:182-192, src/scrapers/germany/ReweScraper.ts:167-241, CLAUDE.md:131-135
Data Models and Configuration
Scrapers are driven by a configuration object that defines URLs, CSS selectors (for HTML scraping), and wait times.
Scraper Configuration
| Property | Type | Description |
|---|---|---|
name | string | Supermarket name (e.g., "REWE", "Auchan") |
baseUrl | string | The root domain of the store |
categories | CategoryConfig[] | List of IDs and URL slugs to scrape |
selectors | Object | CSS selectors for name, price, and image |
waitTimes | Object | Delays for page load and between requests |
maxRetries | number | Number of retry attempts for failed requests |
Sources: src/scrapers/base/BaseScraper.ts:25-30, scripts/test-scrapers.ts:79-90, src/scrapers/germany/ReweScraper.ts:51-68
Extracted Product Schema
The ProductData object is the standard output for all scrapers:
- name: Original product name.
- price: Current numeric price.
- currency: ISO code (e.g., EUR, UAH, TRY).
- isOnSale: Boolean flag for discounted items.
- unit/unitQuantity: Normalized quantity info (e.g., "kg", 1.5).
- externalId: Unique identifier from the supermarket's own system (e.g., SKU).
Sources: src/scrapers/base/BaseScraper.ts:265-276, src/scrapers/ukraine/AuchanUaGraphQLScraper.ts:337-350
Operational Tools
The service includes command-line utilities for developers to manage and test scrapers.
| Command | Description |
|---|---|
npm run scraper:run | Runs all active scrapers in parallel (default concurrency: 3). |
npm run scraper:test | Runs a specific scraper for testing and displays a sample of 10 products. |
npm run scraper:run -- [name] --list-categories | Lists all available categories for a specific supermarket. |
Sources: scripts/run-scraper.ts:10-20, scripts/test-scrapers.ts:12-20
Conclusion
The Scraper Backend Service provides a robust and extensible framework for large-scale grocery price monitoring. By abstracting the complexities of browser automation and anti-bot measures into a common base, the system allows for rapid implementation of new supermarkets while maintaining high data quality through incremental saving and detailed logging.