Feature: SearXNG Integration
Relevant source files
The following files were used as context for generating this wiki page:
- app/Services/SearchService.php
- app/Filament/Resources/ProductResource/Widgets/CreateViaSearchForm.php
- app/Console/Commands/BuildSearchResearch.php
- app/Enums/IntegratedServices.php
- app/Filament/Pages/AppSettingsPage.php
- tests/Feature/Integrations/SearchXNG/SearchResearchTest.php
- resources/views/filament/resources/product-resource/pages/view.blade.php
- README.md
The SearXNG integration allows PriceBuddy users to search for products on the web using a self-hosted SearXNG instance. This feature helps users discover and add new product URLs for price tracking directly within the application. When a search is initiated, PriceBuddy queries the configured SearXNG instance, fetches the results, scrapes each result URL for product information (like price), and caches them. The user can then review these cached results and add them to new or existing products.
The entire process, from fetching search results to scraping individual pages, is handled asynchronously to avoid blocking the user interface.
Sources: README.md, app/Services/SearchService.php, app/Filament/Resources/ProductResource/Widgets/CreateViaSearchForm.php
Configuration
The SearXNG integration must be enabled and configured in the application's settings page before it can be used. The settings are managed in AppSettingsPage and are stored under the integrated_services key with the identifier searxng.
Sources: app/Filament/Pages/AppSettingsPage.php:201-236, app/Enums/IntegratedServices.php:10
The following configuration options are available in the UI:
| Option | Type | Description | Default |
|---|---|---|---|
| Enabled | Toggle | Enables or disables the SearXNG integration feature. | false |
| SearXNG url | string | The full URL of the SearXNG instance, including the /search path. | - |
| Search prefix | string (nullable) | Text to prepend to the product name when searching (e.g., "Buy"). | null |
| Cache duration | Select | How long to keep the parsed search results in the cache. | 14 days |
| How many pages of results to fetch | Select | The number of result pages to fetch from SearXNG for a single query. | 2 pages |
Sources: app/Filament/Pages/AppSettingsPage.php:201-236
Architecture and Data Flow
The integration is composed of several key components: a UI widget for initiating searches, a background job for processing, a core service for fetching and parsing results, and a data model for caching the findings.
High-Level Data Flow
The following diagram illustrates the overall flow from a user initiating a search to the results being available for review.
Core Components
1. CreateViaSearchForm Widget
This Filament widget provides the user interface for initiating a product search. It can be used either to create a new product from a search or to find additional URLs for an existing product.
- On the "Create Product" page: It appears as a standalone form.
- On a product's "View" page: It is embedded within a "Search" tab, pre-filling the search query with the product's title.
When the user submits the form, it dispatches the CacheSearchResults job to process the search asynchronously.
Sources: app/Filament/Resources/ProductResource/Widgets/CreateViaSearchForm.php, resources/views/filament/resources/product-resource/pages/view.blade.php:36-38, tests/Feature/Integrations/SearchXNG/SearchResearchTest.php:120-130
2. CacheSearchResults Job
This job is responsible for executing the search in the background. Its primary role is to instantiate the SearchService and call the build() method with the user's search query. This ensures that the potentially long-running process of fetching and scraping multiple URLs does not impact the user's experience.
Sources: tests/Feature/Integrations/SearchXNG/SearchResearchTest.php:148-154
3. SearchService
This service contains the core logic for the integration. It handles fetching, filtering, normalizing, and processing the search results.
Key methods include:
build(string $searchQuery): The main public method that orchestrates the entire workflow.getRawResults(): Sends an HTTP GET request to the configured SearXNG instance for each page of results up to the configuredmax_pages.filterResults(): Removes irrelevant results from the raw data, such as links to PDF files.normalizeStructure(): Transforms the varied structure of SearXNG results into a consistent format required by PriceBuddy.processResults(): Iterates over the normalized results, scrapes each URL for a price, and persists the findings to theurl_researchtable via theUrlResearchmodel.
Sources: app/Services/SearchService.php
The build process is illustrated below:
4. BuildSearchResearch Command
This is an Artisan command that provides a command-line interface to trigger the search functionality. It accepts a searchQuery argument and, like the background job, calls SearchService::build().
- Command Signature:
app:build-search-research {searchQuery}
Sources: app/Console/Commands/BuildSearchResearch.php:14-25
Asynchronous Search Sequence
The sequence diagram below shows the interaction between the user, the UI, and the backend systems during a search.
Data Storage
Search results are cached in the url_research table. This prevents redundant scraping and allows users to review results before associating them with a product.
UrlResearch Model
The UrlResearch model represents a single cached search result. Based on the SearchService and tests, the key attributes are:
| Attribute | Type | Description |
|---|---|---|
search_query | string | The original search term. |
url | string | The URL of the search result. |
title | string | The title of the search result page. |
snippet | text | A short description from the search result. |
thumbnail | string | URL of a thumbnail image, if available. |
domain | string | The domain of the URL. |
price | decimal | The price found after scraping the URL. Can be null. |
relevance | integer | A relevance score (currently defaults to 0). |
in_progress | boolean | Flag indicating if the search is currently being processed. |
is_complete | boolean | Flag indicating if the search has been completed. |
log | json | A log of actions taken during the scraping process for this URL. |
Sources: app/Services/SearchService.php:153-171, tests/Feature/Integrations/SearchXNG/SearchResearchTest.php:74-83
Summary
The SearXNG integration is a powerful feature for discovering product URLs. By leveraging a self-hosted search engine and an asynchronous processing queue, it provides a seamless way for users to expand their price tracking capabilities without leaving the PriceBuddy application. The system is configurable through the admin settings and provides a clear workflow from search initiation to adding a result to a product.
Sources: README.md, app/Filament/Pages/AppSettingsPage.php:201-207