Guide: Adding a New Supermarket Scraper

Relevant source files

The following files were used as context for generating this wiki page:

• src/scrapers/base/BaseScraper.ts
• src/scrapers/scraperRegistry.ts
• CLAUDE.md
• plan.md
• scripts/test-scrapers.ts
• src/database/migrations/002_create_supermarkets.sql
• src/scrapers/germany/ReweScraper.ts

Adding a new supermarket scraper to the WhereIsLifeCheaper project involves implementing a specialized class that inherits from a common base, configuring it with supermarket-specific selectors, and registering it within the system. The project uses a modular architecture where each scraper is responsible for navigating a specific retail site, extracting product data, and handling site-specific challenges like anti-bot measures or Cloudflare protection.

Sources: CLAUDE.md:12-19, plan.md:19-25

Scraper Architecture

The system follows a Template Method pattern where the BaseScraper defines the skeleton of the scraping process, and concrete implementations provide the specific logic for different supermarkets.

The BaseScraper Class

All scrapers must extend the abstract BaseScraper class. This class provides shared utilities for browser management via Playwright, navigation with retry logic, logging, and error handling.

A diagram showing the inheritance relationship between the abstract BaseScraper and concrete implementations.

Sources: src/scrapers/base/BaseScraper.ts:16-43, plan.md:73-82

Required Implementation Methods

Every new scraper must implement the following four methods:

Method	Description
initialize()	Sets up the browser context, handles initial cookies, or performs location selection (e.g., entering a postal code).
scrapeCategory(category)	Navigates through a specific category, handles pagination, and returns a list of ProductData.
scrapeProductDetails(url)	Extracts deeper metadata from a specific product page if not available on the list view.
cleanup()	Gracefully closes the browser and clears temporary session data.

Sources: CLAUDE.md:52-57, src/scrapers/base/BaseScraper.ts:58-75

Implementation Steps

1. Define Category and Scraper Configurations

Before writing the class, define the categories to be scraped and the technical configuration. Configuration includes selectors for product cards, names, prices, and images.

export const mySupermarketCategories: CategoryConfig[] = [
  { id: 'dairy', name: 'Dairy', url: '/category/dairy' }
];

export const mySupermarketConfig: Partial<ScraperConfig> = {
  name: 'MySupermarket',
  baseUrl: 'https://example.com',
  categories: mySupermarketCategories,
  selectors: {
    productCard: '.item',
    productName: '.item-title',
    productPrice: '.item-price',
    productImage: 'img',
    productUrl: 'a',
  },
  waitTimes: {
    pageLoad: 2000,
    dynamicContent: 1500,
    betweenRequests: 1000,
  }
};

Sources: src/scrapers/germany/ReweScraper.ts:21-65, plan.md:213-228

2. Implement the Scraper Class

The class should be placed in src/scrapers/{country}/. Use the provided utilities in BaseScraper to handle common tasks.

A flowchart illustrating the typical execution flow of a concrete scraper implementation.

Sources: src/scrapers/base/BaseScraper.ts:85-115, src/scrapers/germany/ReweScraper.ts:108-142

3. Register the Scraper

The new scraper must be added to the src/scrapers/scraperRegistry.ts (or equivalent registration point) so that the ScraperFactory can instantiate it by name or ID.

Sources: CLAUDE.md:144-148, plan.md:154-157

4. Database Seed Data

Add the supermarket to the database via the supermarkets table. This entry links the supermarket to a country and specifies the scraper_class name.

INSERT INTO supermarkets (country_id, name, website_url, scraper_class, is_active)
VALUES (1, 'My Supermarket', 'https://example.com', 'MySupermarketScraper', true);

Sources: src/database/migrations/002_create_supermarkets.sql:5-15, plan.md:111-114

Handling Anti-Bot Mechanisms

Many supermarkets use Cloudflare or other bot detection. Implementations can use playwright-extra with the stealth plugin to mitigate these.

• Stealth Plugin: Used in ReweScraper.ts to bypass detection.
• User Agents: Rotate user agents from top-user-agents.
• Proxy Support: BaseScraper automatically applies proxies if configured in environment variables.
• Wait Times: Use randomized jitter between requests to avoid pattern detection.

Sources: src/scrapers/germany/ReweScraper.ts:10-14, src/scrapers/base/BaseScraper.ts:186-215, src/scrapers/base/BaseScraper.ts:246-252

Testing the Scraper

The project provides a dedicated test script to validate scraper logic without running the full pipeline.

# List available scrapers
npx ts-node scripts/test-scrapers.ts --list-scrapers

# Test a specific scraper on a single category
npx ts-node scripts/test-scrapers.ts MySupermarketScraper dairy

This script initializes the scraper, runs the scrapeProductList method, and outputs a sample of 10 products and statistics to the console.

Sources: scripts/test-scrapers.ts:15-30, CLAUDE.md:94-98

Summary

Adding a new scraper requires extending BaseScraper, implementing specific extraction logic, and registering the class. By leveraging the base class's built-in retry logic, proxy handling, and browser management, developers can focus on site-specific DOM traversal and data normalization. Once tested via scripts/test-scrapers.ts, the scraper can be integrated into the daily automated scraping cycle.