Feature: Data Scraping Engine

Relevant source files

The following files were used as context for generating this wiki page:

• app/Services/ScrapeUrl.php
• app/Enums/ScraperStrategyType.php
• app/Enums/ScraperService.php
• app/Services/AutoCreateStore.php
• config/price_buddy.php
• database/seeders/Stores/australia.php
• tests/Unit/Services/ScrapeUrlTest.php
• README.md

The Data Scraping Engine is a core component of PriceBuddy responsible for fetching and parsing product information from various online retailer websites. Its primary function is to extract key product details—specifically title, price, and image—from a given URL. The engine is designed to be flexible, supporting multiple scraping services and data extraction strategies to accommodate a wide range of website structures, from simple static HTML pages to complex JavaScript-rendered sites.

A key feature of the engine is its ability to automatically generate scraping configurations for new, unrecognized stores. By attempting a series of predefined extraction strategies, it can often create a functional store profile without manual intervention, greatly simplifying the process of adding new retailers to the application.

Core Scraping Service

The main entry point for all scraping operations is the ScrapeUrl service. This class orchestrates the process of fetching a URL, parsing its content, and returning the extracted data.

Sources: app/Services/ScrapeUrl.php

Scraping Process Flow

The scrape() method in ScrapeUrl.php follows a distinct sequence of operations, including a retry mechanism to handle transient network issues or page loading failures.

The diagram below illustrates the high-level process flow of the ScrapeUrl service.

Key Operations

1. Store Identification: Before scraping, the service identifies the appropriate Store model by matching the host of the input URL against the domains configured for each store. If no matching store is found, the process is aborted.
   Sources: app/Services/ScrapeUrl.php:178-182
2. Scraper Service Selection: Each store can be configured to use a specific scraper service. The ScrapeUrl service instantiates the appropriate scraper based on the store's settings.
   Sources: app/Services/ScrapeUrl.php:82-96, app/Services/ScrapeUrl.php:157
3. Data Extraction: Once the page content is fetched, the service iterates through the scrape_strategy defined on the Store model for title, price, and image. It uses the scrapeOption method to apply the specified extraction logic (e.g., CSS selector, regex).
   Sources: app/Services/ScrapeUrl.php:167-175
4. Retry Mechanism: If scraping fails (e.g., no title is found), the process will be retried up to a configured maximum number of attempts (max_attempts_to_scrape). Subsequent attempts bypass the cache.
   Sources: app/Services/ScrapeUrl.php:102-116, tests/Unit/Services/ScrapeUrlTest.php:78-87
5. Error Handling: The service includes robust logging and optional UI notifications for errors, such as failing to find a store or missing required fields after scraping.
   Sources: app/Services/ScrapeUrl.php:211-231

Scraping Services and Strategies

The engine's flexibility comes from its support for different backend scraper services and data extraction strategies.

Scraper Services

The ScraperService enum defines the available backend services for fetching web page content.

Service	Value	Description
Http	http	Uses a standard Goutte/Symfony HTTP client. Suitable for server-rendered static HTML sites.
Api	api	Uses an external API, typically a Selenium-based service, to render JavaScript-heavy sites before scraping. The API URL is configured in price_buddy.php.

Sources: app/Enums/ScraperService.php, README.md:16, config/price_buddy.php:14-17

The service to be used is defined in the settings of a Store model. If not specified, it defaults to Http.

Sources: tests/Feature/Models/StoreTest.php:38-48, database/seeders/Stores/australia.php:212-216

Extraction Strategies

The ScraperStrategyType enum defines the methods available for extracting data from the page content.

Strategy	Value	Description
Selector	selector	Uses a CSS selector to find an element. Can also extract a specific attribute (e.g., `meta[name=og:title]
Regex	regex	Uses a regular expression to extract data from the raw HTML content.
Json	json	Extracts data from a JSON structure using dot notation. Useful for API responses or JSON-LD data.
Xpath	xpath	Uses an XPath expression to find an element.

Sources: app/Enums/ScraperStrategyType.php, app/Services/ScrapeUrl.php:189-196

A strategy for a store is defined in the scrape_strategy array, specifying the type and value for each data point (title, price, image).

// Example from database/seeders/Stores/australia.php:11-26
'scrape_strategy' => [
    'title' => [
        'value' => 'title',
        'type' => 'selector',
    ],
    'price' => [
        'value' => '.a-price > .a-offscreen',
        'type' => 'selector',
    ],
    'image' => [
        'value' => '~"hiRes":"(.+?)"~',
        'type' => 'regex',
    ],
],

Sources: database/seeders/Stores/australia.php:11-26

Auto-Create Store Service

When a user provides a URL from a store not yet in the database, the AutoCreateStore service attempts to automatically generate a valid scraping configuration.

Sources: app/Services/AutoCreateStore.php

Auto-Creation Flow

The service fetches the URL and iterates through a list of predefined strategies for title, price, and image. The first strategy that successfully extracts data for a given field is chosen for the new store's configuration.

This diagram shows the process for automatically determining the scraping strategy for a new store.

Default Strategies

The default strategies are defined in config/price_buddy.php. The system tries selectors first, then falls back to regular expressions.

Default Title Strategies

Type	Value
selector	meta[property="og:title"]|content
selector	title
selector	h1

Default Price Strategies

Type	Value	Description
selector	meta[property="product:price:amount"]|content	Open Graph price meta tag
selector	.a-price .a-offscreen	Amazon
selector	[itemProp="price"]|content	Schema.org microdata
selector	.price	Common class name
selector	.product-price, .product-price-value	Common class names
selector	[class^="price"]	Class starts with "price"
selector	[class*="price"]	Class contains "price"
regex	~\"price\"\:\s?\"(.*?)\"~	JSON-like price property
regex	~>\$(\d+(\.\d{2})?)<~	Price inside an HTML tag
regex	~\$(\d+(\.\d{2})?)~	Price anywhere in the text

Default Image Strategies

Type	Value	Description
selector	meta[property="og:image"]|content	Open Graph image meta tag
selector	meta[property="og:image:secure_url"]|content	Open Graph secure image
regex	~\"hiRes\":\"(.+?)\"~	Amazon high-res image
regex	~\"image\"\:\s?\"(.*?\.jpg)\"~	JSON-like image property
regex	~\"image\"\:\s?\"(.*?\.png)\"~	JSON-like image property

Sources: config/price_buddy.php:23-73

Summary

The Data Scraping Engine is a robust and adaptable system for product data acquisition. It combines a primary ScrapeUrl service with a strategic AutoCreateStore feature. By supporting multiple scraping backends (Http, Api) and extraction methods (selector, regex, json, xpath), it can handle a wide variety of target websites. The auto-creation capability, driven by a configurable set of common patterns, significantly enhances the application's usability by allowing users to add new stores with minimal friction. This dual approach of pre-configured and automatically-generated scraping rules makes the engine a cornerstone of the PriceBuddy application.