Background Jobs & Queues
Relevant source files
The following files were used as context for generating this wiki page:
The PriceBuddy application utilizes a background job and queue system to handle long-running tasks asynchronously. This ensures that the user interface remains responsive while operations like fetching product prices and caching search results are performed in the background. The system is built on Laravel's queue functionality and is managed by Supervisor within the Docker environment.
This document outlines the architecture, configuration, and key components of the background job processing system. It covers how jobs are configured, dispatched, and executed by queue workers.
System Architecture
The background processing system relies on several components working together: Laravel's queue system, specific job classes that define the work to be done, a queue driver (database by default), and Supervisor to keep the queue workers and scheduler running.
The following diagram illustrates the high-level flow of job processing in the application.
This diagram shows the general flow from job dispatch to execution by a worker process managed by Supervisor.
Sources: config/queue.php:15, docker/php/supervisord.conf:10-17, app/Jobs/UpdateProductPricesJob.php
Queue Configuration
The queue system is configured in config/queue.php. By default, PriceBuddy uses a database-driven queue.
Sources: config/queue.php
Default Connection
The application is configured to use the database queue connection by default. This means that jobs dispatched to the queue will be stored in a database table.
'default' => env('QUEUE_CONNECTION', 'database'),
Sources: config/queue.php:15
The configuration for the database connection is as follows:
| Parameter | Environment Variable | Default Value | Description |
|---|---|---|---|
driver | - | database | The queue driver to use. |
connection | DB_QUEUE_CONNECTION | null | The database connection to use. |
table | DB_QUEUE_TABLE | jobs | The database table to store jobs. |
queue | DB_QUEUE | default | The default queue name. |
retry_after | DB_QUEUE_RETRY_AFTER | 90 | Seconds to wait before retrying a job. |
after_commit | - | false | Dispatch jobs only after DB transactions are committed. |
Sources: config/queue.php:29-36
Failed Jobs
Configuration for handling failed jobs is also defined in config/queue.php. By default, failed jobs are logged to a database table using UUIDs.
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
'database' => env('DB_CONNECTION', 'sqlite'),
'table' => 'failed_jobs',
],
Sources: config/queue.php:88-92
Job Processing and Management
Within the Docker environment, supervisord is responsible for managing the queue worker process. This ensures that jobs are continuously processed.
Sources: docker/php/supervisord.conf, docker/php/start-app.sh:28
Supervisor Configuration
The supervisord.conf file defines a program named laravel-worker which runs the php artisan queue:work command. This command starts a worker that processes jobs from the default queue.
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /app/artisan queue:work --sleep=3 --tries=3
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/worker.log
stopwaitsecs=3600
Sources: docker/php/supervisord.conf:10-22
The application is started via the /start-app.sh script, which concludes by launching supervisord.
# /start-app.sh
# ... (setup steps) ...
# Start supervisor that handles cron and apache.
supervisord -c /etc/supervisor/conf.d/supervisord.conf
Sources: docker/php/start-app.sh:27-28
The following diagram shows how Supervisor manages the application's background processes.
This diagram illustrates how supervisord orchestrates the web server, cron, and queue worker processes.
Sources: docker/php/supervisord.conf, docker/php/start-app.sh:28
Key Job Implementations
The application defines several key jobs for its background operations.
| Job Class | Description |
|---|---|
UpdateAllPricesJob | A scheduled job that iterates through all products and dispatches individual UpdateProductPricesJob for each one. |
UpdateProductPricesJob | Fetches the latest price for a single product from its associated URLs. |
CacheSearchResults | Performs a search via the configured SearXNG instance and caches the results. |
UpdateAllPricesJob
This job is responsible for initiating the price update process for all products. It chunks the products and dispatches an UpdateProductPricesJob for each one.
Sources: app/Jobs/UpdateAllPricesJob.php
UpdateProductPricesJob
This is the core job for fetching prices. It is dispatched for each product that needs a price update.
Execution Flow
The handle method orchestrates the price fetching process for a single product.
- Logs the start of the price fetch if logging is enabled.
- Calls the
updatePrices()method on theProductmodel. - Logs the success or failure of the operation.
- If the scrape fails, it notifies the product's owner via a
ScrapeFailNotification. - Pauses execution for a configured number of seconds (
sleep_seconds_between_scrape) before completing, to avoid rate-limiting.
Sources: app/Jobs/UpdateProductPricesJob.php:20-41
The sequence diagram below illustrates the process.
This diagram shows the sequence of operations within the UpdateProductPricesJob.
Sources: app/Jobs/UpdateProductPricesJob.php:20-41
CacheSearchResults
This job handles the asynchronous caching of search results from a SearXNG instance. It is designed to be robust, with built-in retry logic.
Job Properties
- Tries:
3 - Backoff Strategy:
[30, 60, 120]seconds - Timeout:
3600seconds - Unique Lock:
600seconds (10 minutes) to prevent duplicate jobs for the same query.
Sources: app/Jobs/CacheSearchResults.php:16-29
Execution Flow
The handle method uses the SearchService to perform the search and cache the results. Any exceptions are caught, logged, and the job is failed, allowing for retries based on the backoff strategy.
// app/Jobs/CacheSearchResults.php:39-49
public function handle(): void
{
try {
$this->searchService
->log('Job dispatched')
->build($this->searchQuery);
} catch (Exception $e) {
logger()->error('Search results caching failed', [
'query' => $this->searchQuery,
'error' => $e->getMessage(),
]);
$this->fail($e);
}
}
Sources: app/Jobs/CacheSearchResults.php:39-53
The relationship between the job and the service is shown below.
This class diagram shows that CacheSearchResults uses an instance of SearchService to perform its work.
Sources: app/Jobs/CacheSearchResults.php:31-34, 42
Summary
The background job and queue system is a critical part of PriceBuddy, enabling it to perform essential but time-consuming tasks like web scraping without degrading the user experience. By leveraging Laravel's queue system with a database driver and managing processes with Supervisor, the application achieves a reliable and scalable architecture for asynchronous processing. The specific job implementations for updating prices and caching searches are central to the application's core functionality.