Scraper Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Scraper Architecture is the foundational component of the WhereIsLifeCheaper project, designed to automate the daily collection of grocery price data from various supermarkets across multiple countries including Turkey, Germany, Ukraine, and Malaysia. The system employs a modular, object-oriented design that allows for diverse scraping strategies—ranging from browser-based automation to direct API and GraphQL consumption—while maintaining a unified interface for data processing and storage.
Sources: plan.md:1-15, CLAUDE.md:3-8, README.md:5-15
Core Component Hierarchy
The system follows a template method pattern centered around an abstract base class. This ensures consistent logging, error handling, and browser management across all supermarket implementations.
BaseScraper Abstract Class
The BaseScraper class provides the lifecycle management for every scraper. It handles Playwright browser initialization, proxy configuration, and common utility methods like extractText and navigateToUrl.
- Browser Management: Handles launching Chromium via Playwright, setting up browser contexts, and cleaning up resources.
- Resilience: Implements retry logic with exponential backoff and screenshot capture on failures for debugging.
- Statistics Tracking: Monitors
productsScraped,productsFailed, and execution duration.
Sources: src/scrapers/base/BaseScraper.ts:16-54, CLAUDE.md:46-51
Scraper Class Hierarchy
The following diagram illustrates the relationship between the base architecture and specialized implementations.
The diagram shows how specialized scrapers extend the base class to implement specific extraction logic while inheriting core lifecycle methods.
Sources: src/scrapers/base/BaseScraper.ts:16-43, src/scrapers/ukraine/AuchanUaGraphQLScraper.ts:108-115, src/scrapers/germany/ReweScraper.ts:68-80
Scraping Strategies
The architecture supports three primary data extraction methods based on the target supermarket's technical infrastructure.
| Strategy | Implementation | Description |
|---|---|---|
| DOM Automation | Playwright (Chromium) | Navigates the UI, handles pagination buttons, and parses HTML elements using CSS selectors. Used for standard websites like SPAR Albania. |
| REST API | Fetch / Playwright Request | Bypasses the UI to request JSON data directly from backend endpoints. Used by WinMart (Vietnam) and Knuspr (Germany). |
| GraphQL | HTTPS POST | Sends structured queries to GraphQL endpoints for high-performance extraction without a browser. Used by Auchan Ukraine (Express). |
| Stealth Mode | Playwright Extra + Stealth | Uses persistent contexts and human-like interactions to bypass Cloudflare and Turnstile challenges. Used by REWE Germany. |
Sources: src/scrapers/albania/SparAlbaniaScraper.ts:58-65, src/scrapers/germany/KnusprScraper.ts:85-95, src/scrapers/ukraine/AuchanUaGraphQLScraper.ts:108-114, src/scrapers/germany/ReweScraper.ts:98-110
Execution Lifecycle
The scraping process is orchestrated by a ScraperService (or manual test scripts) and follows a strictly defined sequence to ensure data integrity and avoid rate limiting.
Scraping Sequence
This diagram depicts the standard flow from initialization through to data persistence via the onPageScraped callback.
Sources: src/scrapers/base/BaseScraper.ts:74-110, scripts/run-scraper.ts:40-60, plan.md:144-165
Resilience and Anti-Detection
To handle sophisticated anti-bot measures (e.g., Cloudflare, rate limiting), the architecture includes specialized components for stealth and recovery.
Anti-Detection Features
- Stealth Plugin: Integration with
puppeteer-extra-plugin-stealthviaplaywright-extrato mask automation signatures. - User-Agent Rotation: Utilizes
top-user-agentsto provide realistic, diverse browser headers. - Proxy Support: Support for per-supermarket proxy configurations, including authentication.
- Jitter and Delays: Randomized wait times between requests to mimic human behavior.
Sources: src/scrapers/base/BaseScraper.ts:121-135, src/scrapers/germany/ReweScraper.ts:39-45, src/scrapers/base/BaseScraper.ts:199-215
Error Recovery Flow
The flow details the automated recovery and diagnostic steps taken when a scraper encounters obstacles.
Sources: src/scrapers/base/BaseScraper.ts:219-232, src/scrapers/base/BaseScraper.ts:291-305
Configuration and Management
Scrapers are configured via a centralized ScraperConfig object, which defines behavior parameters and supermarket-specific metadata.
// Example from scripts/test-scrapers.ts
const config: ScraperConfig = {
supermarketId: "test-id",
name: "SupermarketName",
baseUrl: "https://example.com",
categories: [...],
selectors: {
productCard: ".item",
productPrice: ".price",
},
waitTimes: { pageLoad: 3000, betweenRequests: 1000 },
maxRetries: 3,
concurrentPages: 1
};
Sources: scripts/test-scrapers.ts:75-95, src/scrapers/base/BaseScraper.ts:31-36
Orchestration Methods
- Manual Testing: Developers use
scripts/test-scrapers.tsto validate individual scrapers or specific categories using flags like--list-categories. - Production Runs:
scripts/run-scraper.tshandles multi-scraper execution with concurrency control (default: 3 parallel runs). - Automated Scheduling: Managed via GitHub Actions or EventBridge (AWS), triggering runs daily at 3 AM UTC.
Sources: scripts/test-scrapers.ts:12-25, scripts/run-scraper.ts:21-35, terraform/main.tf:305-312
Summary
The Scraper Architecture serves as a robust engine for the project's data pipeline. By abstracting core browser logic into the BaseScraper and allowing specialized implementations for REST, GraphQL, and Stealth browsing, the system efficiently manages the complexities of scraping diverse international grocery platforms while ensuring data consistency and system resilience.
Sources: plan.md:200-210, CLAUDE.md:120-130