Background Jobs & Workers

Relevant source files

The following files were used as context for generating this wiki page:

• src/hooks/useJobPolling.ts
• src/workers/analysis.worker.ts
• DEPLOYMENT_CHECKLIST.md
• GEMINI.md
• MAJOR_FIX_UPDATE_SUMMARY.md
• FINAL_REPORT_TASK_COMPLETION.md
• public/sw.js
• src/serviceWorker.ts

Introduction

The Background Jobs & Workers system in BMSview provides a mechanism for handling long-running, asynchronous tasks, specifically the AI-powered analysis of BMS screenshots. This system was primarily designed for bulk uploads, where processing each image immediately would lead to a poor user experience. The architecture involves the frontend initiating analysis jobs, the backend processing them asynchronously, and the frontend polling for completion status.

It is important to note that this asynchronous, job-based flow is considered a legacy feature and is being deprecated. The project is moving towards a local-first sync architecture and a synchronous analysis flow (/api/analyze?sync=true) for a faster and more resilient user experience. The legacy job functions have been archived to prevent conflicts with the new architecture.

Sources: GEMINI.md, MAJOR_FIX_UPDATE_SUMMARY.md

System Architecture

The asynchronous analysis flow is a classic client-server pattern involving job queuing and status polling. The process begins when the user uploads images for analysis. The frontend prepares the data and makes a request to the backend, which creates a job and returns a job ID. The frontend then uses this ID to periodically ask the backend for the job's status until it is complete.

The following diagram illustrates the high-level data flow for the asynchronous job processing system.

Frontend Implementation

Client-Side Pre-processing (analysis.worker.ts)

Before an analysis job is created, the frontend uses a Web Worker to perform pre-processing on the client side. The primary responsibility of analysis.worker.ts is to calculate a SHA-256 hash of the image content. This hash is used by the backend for deduplication, allowing the system to quickly return existing analysis results for identical screenshots without re-processing them. This offloads computationally intensive work from the main UI thread, keeping the application responsive.

Sources: src/workers/analysis.worker.ts

Job Polling (useJobPolling.ts)

The useJobPolling custom React hook is the core frontend component for managing asynchronous jobs. Once a job is submitted and job IDs are received, this hook takes over to monitor their status.

Key Features:

• State Management: Manages the polling state (isPolling) and the statuses of all monitored jobs (jobStatuses).
• Exponential Backoff: To avoid overwhelming the server, the hook implements an exponential backoff strategy. The polling interval starts small and increases with each subsequent request for a given job.
• Lifecycle Management: Polling starts when jobIds are provided and stops automatically once all jobs have reached a terminal state ('Completed' or 'Failed').
• Callbacks: Provides onUpdate and onComplete callbacks to allow parent components to react to status changes and final completion.

The sequence diagram below shows the interaction between a React component, the useJobPolling hook, and the backend API.

Service & Retry Logic (clientService.ts)

For certain long-running background tasks, such as generating detailed insights, the initial request might time out. The clientService.ts file contains logic to handle these scenarios gracefully. If a generate-insights request returns a 408 Timeout status with a jobId, the client service automatically initiates a retry-and-resume mechanism. It will poll a status endpoint with the provided jobId up to a maximum number of attempts, providing the user with progress updates.

This retry logic is distinct from but complementary to the useJobPolling hook, targeting specific API endpoints that have a synchronous-first, asynchronous-fallback pattern.

Sources: FINAL_REPORT_TASK_COMPLETION.md

Service Worker (sw.js & serviceWorker.ts)

The repository includes a service worker (public/sw.js) and a registration script (src/serviceWorker.ts). The service worker is configured using Workbox and is responsible for precaching application assets for offline availability and improved performance. It is a standard Progressive Web App (PWA) feature and is not directly involved in the background analysis job processing or polling logic. Its "background" role pertains to asset caching and network request interception, not application-specific job execution.

Sources: public/sw.js, src/serviceWorker.ts

Backend Implementation (Netlify Functions)

The backend logic for the asynchronous job system is implemented as a set of serverless Netlify Functions. These functions were part of the original architecture but have since been archived.

Sources: MAJOR_FIX_UPDATE_SUMMARY.md

Job Creation (/api/analyze)

This function served as the entry point for the asynchronous workflow.

• It received a request containing image data.
• It created a job record in the jobs collection in MongoDB with a status of Queued.
• It returned the jobId to the client.
• It asynchronously invoked the process-analysis function to start the actual work.

Sources: DEPLOYMENT_CHECKLIST.md, GEMINI.md

Job Processing (/api/process-analysis)

This function was the background worker.

• It was invoked by the analyze function.
• It would update the job's status in the database to Processing.
• It performed the core AI analysis on the provided data.
• Upon completion, it updated the job's status to Completed and stored the result, or to Failed if an error occurred.

Sources: DEPLOYMENT_CHECKLIST.md, MAJOR_FIX_UPDATE_SUMMARY.md

Job Status Retrieval (/api/get-job-status)

This function was the endpoint called by the useJobPolling hook.

• It accepted a list of jobIds in a POST request.
• It queried the jobs collection in MongoDB for the statuses of the requested jobs.
• The response time of this function was optimized by using database indexes on the jobs collection.
• It returned the current status of each job, which the frontend used to update the UI.

Sources: DEPLOYMENT_CHECKLIST.md

Data & State Management

Job Status Lifecycle

Each job progresses through a simple, defined lifecycle. The status is tracked in the jobs collection in the database.

This state diagram shows the possible transitions for a job.

Database Schema (jobs collection)

The jobs collection in MongoDB was used to store the state and results of analysis jobs. While the exact schema is not fully detailed, the DEPLOYMENT_CHECKLIST.md mentions key indexes that imply the existence of certain fields.

Field	Type	Description
id	String	The unique identifier for the job.
status	String	The current status of the job (e.g., 'Queued', 'Processing', 'Completed'). Indexed for fast lookups.
created	Date	Timestamp when the job was created. Indexed for TTL and sorting.
result	Object	The analysis result, stored upon completion.
error	String	Error message, stored if the job fails.

Database Indexes:
To ensure efficient querying, especially for the get-job-status endpoint, several indexes were created on the jobs collection. These typically included indexes on id, status, and created.

Sources: DEPLOYMENT_CHECKLIST.md

Conclusion

The background jobs and workers system was an essential part of BMSview's initial architecture, enabling non-blocking, asynchronous processing of analysis tasks. It comprised a frontend with a polling hook (useJobPolling) and client-side workers, a set of backend serverless functions for job management, and a MongoDB collection for state persistence. However, as the project has evolved, this entire asynchronous flow has been deprecated in favor of a more modern, efficient, and resilient synchronous and local-first architecture.

Sources: GEMINI.md, MAJOR_FIX_UPDATE_SUMMARY.md