Error Handling & Retries
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The WhereIsLifeCheaper project employs a multi-layered error handling and retry system designed to ensure high resilience during complex web scraping operations and API service delivery. Given the volatile nature of scraping supermarket websites across multiple countries, the system utilizes specialized utility functions for exponential backoff, modular logging via Winston, and defensive coding patterns within the scraper base classes to handle network fluctuations, anti-bot measures, and database connectivity issues.
The core strategy focuses on "Error Resilience," which incorporates retry logic for network requests, comprehensive logging for audit trails, and automatic status tracking for every scraping run. This ensures that transient failures do not compromise the integrity of the daily price data collection process.
Sources: plan.md:23-26, src/scrapers/base/BaseScraper.ts:16-25
Retry Logic Architecture
The project implements a centralized retry mechanism in src/utils/retry.ts. This utility is a generic wrapper that can execute any asynchronous function and automatically retry it upon failure based on configurable parameters.
Exponential Backoff and Jitter
To prevent overwhelming external supermarket servers and to bypass basic rate limiting, the retry utility supports exponential backoff. It also incorporates random jitter to avoid pattern detection by anti-bot systems.
The flow above illustrates how the retry utility handles failures by calculating delays and re-attempting execution until the maximum threshold is reached.
Sources: src/utils/retry.ts:1-25, src/scrapers/base/BaseScraper.ts:210-220
Scraper Navigation Retries
The BaseScraper class utilizes this retry logic specifically for URL navigation. This handles common network timeouts or transient DNS issues encountered when accessing international supermarket domains.
| Parameter | Type | Default/Config Reference | Description |
|---|---|---|---|
maxRetries | number | config.maxRetries (typically 3) | Maximum number of attempts before failing |
initialDelay | number | 2000ms (for navigation) | Starting delay before the first retry |
timeout | number | config.scraper.timeout | Playwright page navigation timeout |
Sources: src/scrapers/base/BaseScraper.ts:192-212, src/scrapers/germany/ReweScraper.ts:39-41
Logging and Monitoring
Error visibility is managed through a sophisticated logging setup using Winston. The system segregates logs into specific components (Scrapers, API, Cron) and supports both local file storage and Google Cloud Logging for production environments.
Specialized Loggers
The application defines multiple logger instances to separate concerns:
- scraperLogger: Captures details about product extraction, selector failures, and navigation issues.
- apiLogger: Records HTTP request/response cycles and server-side exceptions.
- cronLogger: Tracks the scheduling and execution of daily jobs.
Sources: src/utils/logger.ts:120-135, src/api/server.ts:10-15
Scraper Run Auditing
Scrapers track their own internal statistics, which are eventually reported to the scrape_logs table in PostgreSQL. This table stores the status, duration, and error counts for each run.
The BaseScraper maintains an array of ScrapeError objects. Each error captures the specific product URL that failed, the error message, and the stack trace for later analysis.
Sources: src/scrapers/base/BaseScraper.ts:25-30, plan.md:66-68
Defensive Scraping Patterns
Specific scrapers, such as the ReweScraper, implement advanced error handling to manage environment-specific challenges like Cloudflare protection.
Cloudflare & Anti-Bot Handling
The system uses the following techniques to minimize errors caused by bot detection:
- Stealth Browser: Integration with
playwright-extraandpuppeteer-extra-plugin-stealthto evade fingerprinting. - Turnstile Solving: Logic to detect and click Cloudflare Turnstile checkboxes within iframes.
- Screenshot Capture: Automatic screenshots are taken when challenges are detected or when extraction fails, facilitating manual debugging.
Sources: src/scrapers/germany/ReweScraper.ts:80-120, src/scrapers/base/BaseScraper.ts:244-260
API Error Management
In the Express API server, error handling is centralized via middleware. In development, the full error stack is returned to the client, while in production, a generic "Internal Server Error" message is sent to avoid leaking sensitive architecture details.
Sources: src/api/server.ts:133-140
Conclusion
The Error Handling & Retries system in WhereIsLifeCheaper is built on the principle of graceful degradation. By combining automated retries for transient network issues with detailed logging and state tracking for permanent failures, the system maintains high data availability. The use of structured logging and run auditing allows developers to quickly identify and fix selector changes or anti-bot updates that would otherwise disrupt the daily pricing pipeline.