Background Jobs & Derived Data
Relevant source files
The following files were used as context for generating this wiki page:
Background jobs and derived data management in PRKS ensure that the canonical research library remains synchronized with searchable indexes, thumbnails, and offline caches. The system distinguishes between canonical state (SQLite database, PDFs, and managed person files) and derived state (text indexes, research indexes, and thumbnails) which the application can rebuild from canonical sources.
PRKS uses a local-first approach where operations are enqueued as durable intents. These intents are processed in the background to maintain consistency across the server and various client-side caches.
Sources: README.md:144-150, backend/AGENTS.md:112-118
Derived Data Lifecycle
Derived data components are disposable and rebuildable. This architecture simplifies backup and recovery because only canonical data requires persistent protection.
Categories of Derived Data
The system categorizes post-commit cleanup and data maintenance into four primary owners, each with a specific recovery model.
| Category | Storage Component | Recovery Owner | Maintenance Frequency |
|---|---|---|---|
| Text Index | prks_text_index.db | text_index.reconcile_all() | At startup |
| Research Index | prks_research_index.db | PRKSResearchIndex.reconcile_all() | At startup |
| Thumbnails | thumbs/ | prune_orphan_pdf_thumbnails() | At startup |
| Managed PDF | pdfs/ | retry_pending_pdf_cleanup() | Startup, after delete, after restore |
Sources: backend/AGENTS.md:113-118, backend/AGENTS.md:162-171
Derived Data Maintenance Flow
The maintenance flow ensures that stale rows or orphaned files do not persist after the canonical records are modified or deleted.
Sources: backend/AGENTS.md:162-171
Background Synchronization & Durable Jobs
The system processes background tasks through a sync runtime that manages durable operations. This ensures that even during connectivity interruptions, user intents are eventually synchronized with the canonical database.
Sync Runtime Components
The synchronization system consists of a store for durable operations, a runtime coordinator, and specific handlers for different entity families.
- Durable Operations: Records of intent stored in IndexedDB (e.g.,
prks-local-v1). - Sync Runtime: Coordinates the execution of enqueued operations when the environment is online.
- Reconciliation: The process of updating the disposable read cache after a background job completes.
Sources: frontend/js/sync-runtime.js:12-35, frontend/js/offline-runtime.js:45-60
Background Job Execution Sequence
When an operation is enqueued, the background runtime attempts to push the change to the server.
Managed PDF Cleanup
Managed PDFs require a specific durable state for cleanup because deleting a works row destroys the only reference to the physical file. PRKS uses the pending_pdf_cleanup table (schema 16) to track these jobs.
Cleanup Safety Rules
- Reference Verification: The system re-checks the live catalogue immediately before any retry deletion to ensure no new record references the file.
- Transactional Integrity: The reference check and the claim retirement occur within a single transaction to prevent race conditions during Work creation.
- Rotation Selection: The system selects claims for retry based on
last_attempt_at, prioritizing never-attempted claims to prevent "poison" claims from stalling the queue.
Sources: backend/AGENTS.md:183-205
Rebuildable Components
The system provides manual maintenance tools for rebuilding specific derived components if they become corrupt or inconsistent.
Text Index Management
The text index (prks_text_index.db) is entirely derived and must not be included in backups. If schema corruption occurs, the application repairs it by recreating the entire index from the main database. Normal startup avoids full FTS5 integrity checks unless the index is already marked as suspect.
Sources: backend/AGENTS.md:144-158
Database Migrations
PRKS handles database schema upgrades as synchronous background tasks at startup. Migrations are transactional and version-ordered.
| Field | Requirement |
|---|---|
| Versioning | Increment LATEST_SCHEMA_VERSION |
| Execution | Run via backend/db_migrations.py |
| Fresh DB | Final state must match backend/db_schema.sql |
Sources: README.md:131-137, backend/AGENTS.md:218-225, backend/db_migrations.py
Conclusion
The PRKS background job and derived data architecture enforces a strict separation between permanent research data and transient indices. By using durable intents for background synchronization and specialized cleanup recovery models for physical files, the system maintains data integrity while providing high performance for search and navigation. Derived components like text indexes and thumbnails are treated as rebuildable assets, ensuring library portability and resilient recovery from crashes.