Grocery Scraping Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Grocery Scraping Engine is a modular, multi-country data extraction system designed to automate the collection of grocery product information and pricing from various international supermarket chains. It serves as the data ingestion layer for the "WhereIsLifeCheaper" project, tracking daily prices to facilitate cross-country cost-of-living comparisons.
The engine utilizes Playwright for browser automation and is built on a provider-agnostic architecture. This allows it to handle diverse supermarket web structures—ranging from standard HTML server-side rendered pages to complex single-page applications (SPAs) and REST API-backed storefronts—while maintaining a consistent internal data format.
Sources: CLAUDE.md:1-24, plan.md:1-20
Core Architecture
The engine follows a hierarchical structure where a centralized ScraperService orchestrates various concrete scraper implementations through a factory-like registry.
Component Hierarchy
- Orchestrator (
ScraperService): Manages the execution lifecycle, database persistence, and concurrency of scraping jobs. - Base Layer (
BaseScraper): An abstract class providing shared utilities for browser management, network retries, and error logging. - Concrete Scrapers: Supermarket-specific implementations (e.g.,
ReweScraper,VoliScraper) that handle unique DOM selectors or API endpoints. - Registry (
scraperRegistry): A lookup system that maps supermarket identifiers to their respective scraper classes and default configurations.
Sources: src/scrapers/base/BaseScraper.ts:13-30, src/services/ScraperService.ts, src/scrapers/scraperRegistry.ts
System Data Flow
The following diagram illustrates how a scraping request moves from the service layer through the browser automation layer to the database.
The diagram shows the orchestration of a scraping run, highlighting the separation between orchestration logic and implementation details.
Sources: src/scrapers/base/BaseScraper.ts:77-108, scripts/run-scraper.ts:30-80
Scraper Implementation Details
All scrapers must extend the BaseScraper class and implement a core set of lifecycle methods.
BaseScraper Features
The BaseScraper provides a robust foundation for web scraping by handling common "plumbing" tasks:
- Browser Management: Lifecycle of
BrowserandPageobjects, including user-agent rotation. - Resilience: Built-in retry logic with exponential backoff for navigation and extraction tasks.
- Debugging: Automatic screenshot capture upon failure.
- Stealth: Integration with proxy configurations and random jitter between requests to avoid rate limiting.
Sources: src/scrapers/base/BaseScraper.ts:140-275
Extraction Strategies
Scrapers in the engine use three primary strategies for data extraction:
| Strategy | Description | Example Implementation |
|---|---|---|
| DOM Scraping | Uses CSS selectors to find elements in the HTML. | VoliScraper |
| API Interception | Monitors network traffic to capture JSON responses from internal APIs. | LotussApiScraper |
| Direct API | Directly queries the supermarket's REST endpoints using browser cookies. | MercadonaScraper, WinMartScraper |
Sources: src/scrapers/montenegro/VoliScraper.ts:130-180, src/scrapers/spain/MercadonaScraper.ts:175-220, src/scrapers/malaysia/LotussApiScraper.ts:165-190
Anti-Bot and Stealth Mechanisms
To maintain high availability and prevent blocking, the engine implements several stealth techniques, particularly evident in the ReweScraper implementation.
Stealth Configuration
The engine leverages playwright-extra with the stealth plugin to mask browser fingerprints. It also employs persistent browser contexts to maintain sessions across requests, which is critical for sites protected by Cloudflare.
The diagram outlines the initialization sequence used to bypass advanced bot detection.
Sources: src/scrapers/germany/ReweScraper.ts:70-150, src/scrapers/base/BaseScraper.ts:175-195
Proxy Management
Proxies are configured per supermarket in the BaseScraper. The system checks config.scraper.proxyConfig for a key matching the supermarket name (e.g., "rewe") and applies the proxy at the browser launch level.
Sources: src/scrapers/base/BaseScraper.ts:153-172
Execution and Monitoring
The scraping process is triggered via the ScraperService, which can run scrapers individually or in parallel.
Scraping Lifecycle Sequence
This sequence ensures that products are saved incrementally to the database, preventing data loss if a long-running scrape fails mid-way.
Sources: src/scrapers/base/BaseScraper.ts:77-108, src/services/ScraperService.ts, scripts/run-scraper.ts:35-50
Monitoring and Logs
Each run generates detailed statistics and logs:
- scrape_logs table: Stores status (
running,success,failed), products scraped, duration, and error messages. - Winston Logger: Provides prefixed logs (e.g.,
[Rewe|RunID]) for real-time monitoring. - Admin Dashboard: A React-based interface for triggering scrapers manually and viewing recent logs.
Sources: src/scrapers/base/BaseScraper.ts:32-40, frontend/src/pages/admin/Scrapers.tsx:150-240
Summary
The Grocery Scraping Engine provides a highly resilient and extensible framework for international grocery data collection. By abstracting the complexities of browser automation and anti-bot bypass into a BaseScraper, the system allows for rapid development of new scrapers while ensuring consistent data quality and operational monitoring.