File Processing Queue
Relevant source files
The following files were used as context for generating this wiki page:
The File Processing Queue (also referred to as "Files for Processing") manages the recursive ingestion and staging of PDF files from the local filesystem into the PRKS library. It provides an "inbox" workflow where files are scanned from a specific directory, enriched with staged metadata, and eventually imported as library Works. Files in this queue remain isolated from the main library search and research graph until the import process is finalized.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:72-74, backend/api_contract/openapi.py:465-467
System Architecture
The system uses a decoupled architecture with a Vue 3 frontend for user interactions and a Python backend for filesystem operations and metadata persistence. The backend defines strict API contracts using Pydantic models to ensure wire-format consistency.
Data Flow Overview
The following diagram illustrates the lifecycle of a file from the processing directory to the library.
The diagram shows how files transition from the local disk through the staging UI into the canonical library.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:72-74, backend/api_contract/openapi.py:534-560
Backend API Contract
The backend handles file discovery, metadata staging, and final import through a dedicated set of HTTP endpoints. It uses Pydantic-generated schemas for validation.
API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/processing-files | GET | Lists inbox files. Use rescan=true to reconcile the disk folder. |
/api/processing-files/{id} | PATCH | Saves staged metadata (folder, tags, roles, status) for an inbox file. |
/api/processing-files/{id}/import | POST | Imports the staged file into the library as a new Work. |
| Sources: backend/api_contract/openapi.py:495-585 |
Processing File Schema
The ProcessingFile data model tracks the file's state on disk and its staged metadata.
| Field | Type | Description |
|---|---|---|
id | string | Unique identifier for the processing file. |
path | string | Relative path within the inbox directory. |
staged_title | string | Proposed title for the library Work. |
staged_folder_id | string (optional) | Destination folder for the imported file. |
staged_tags | array | List of ProcessingFileTag objects. |
staged_roles | array | List of ProcessingFileRole objects (e.g., Authors). |
status | string | Enumerated status: pending, missing, or failed. |
| Sources: backend/api_contract/processing_files.py:1-40, backend/api_contract/openapi.py:504-510 |
Frontend Implementation
The frontend is implemented as a Vue 3 feature located in frontend-app/src/features/processing/. It utilizes a projection model to handle large sets of files efficiently.
Component Structure
- ProcessingFilesRoute.vue: The main container for the processing view. It manages scanning states, pagination, and resource lifecycle.
- ProcessingFileCard.vue: Renders individual file details, including a PDF preview iframe and metadata inputs for tags, people/roles, and destination folders.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:1-68, frontend-app/AGENTS.md:65-68
Pagination Logic
To maintain UI responsiveness, the queue displays files in pages defined by PROCESSING_PAGE_SIZE.
The sequence shows the transition from backend scanning to incremental frontend rendering.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:24-45
Operations and Lifecycle
Metadata Staging
You can update individual fields of a processing file before importing. The backend validates role types, person IDs, and folder IDs during the PATCH request. The ProcessingFileUpdateRequest model permits partial updates, where sending an empty object {} changes nothing.
Sources: backend/api_contract/openapi.py:521-550
Resource Management
The processing feature manages external resources like preview iframes and window resize listeners. These are attached when the route is mounted and released during the onBeforeUnmount lifecycle hook to prevent memory leaks.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:64-70, frontend-app/AGENTS.md:66-68
Security and Isolation
Files in the processing queue are not indexed by the library's FTS5 text search and do not appear in the research graph. This isolation allows for metadata refinement without polluting the canonical research library.
Sources: frontend-app/src/features/processing/ProcessingFilesRoute.vue:72-74, backend/AGENTS.md:83-90
Summary
The File Processing Queue serves as a critical bridge between unmanaged local storage and the structured PRKS research environment. It combines recursive filesystem scanning with a rich metadata staging interface, ensuring that only properly categorized and annotated research material enters the canonical library.