Data Synchronization & Background Jobs
Relevant source files
The following files were used as context for generating this wiki page:
The Data Synchronization and Background Jobs system is responsible for maintaining consistency between the application's local database and external on-chain data sources. It facilitates incremental updates of prediction market events, resolutions, and sports data, while also managing long-running background tasks such as automated on-chain event deployment.
The system architecture relies on a series of API endpoints secured by a shared secret, which are triggered by external schedulers like Kubernetes CronJobs or PostgreSQL's pg_cron extension. Each synchronization task is designed to be idempotent and includes locking mechanisms to prevent concurrent execution of the same job.
Sources: src/lib/sync/cron-route.ts:1-20, infra/kubernetes/cronjobs.yaml:1-20
Core Synchronization Framework
The platform uses a standardized utility framework to handle periodic tasks. This framework provides authentication, concurrency control, and status tracking for all background jobs.
Cron Route Handler
The handleCronRoute function serves as the primary entry point for all synchronization endpoints. It performs the following tasks:
- Authentication: Validates the
Authorization: Bearer <CRON_SECRET>header against theCRON_SECRETenvironment variable. - Error Handling: Catches exceptions during execution and provides a structured JSON error response.
- Logging: Logs job failures for observability.
Sources: src/lib/sync/cron-route.ts:31-64
Concurrency and Locking
To prevent multiple instances of a sync job from running simultaneously (which could lead to race conditions or duplicate data), the system implements a distributed lock using the subgraph_syncs database table.
The locking mechanism includes a staleMs threshold (defaulting to 15 minutes) to recover from jobs that crashed without releasing their lock.
Sources: src/lib/sync/cron-route.ts:66-105
Event Synchronization
The event sync process pulls new and updated prediction markets from an external PnL (Profit and Loss) subgraph.
Incremental Fetching Logic
The synchronization is incremental, using a cursor-based approach. The cursor, consisting of a conditionId and updatedAt timestamp, is stored in the subgraph_syncs table. On the first run, the system performs an initial lookback (defaulting to 14 days) to bootstrap the database.
Sources: src/app/api/sync/events/route.ts:100-145, src/app/api/sync/events/route.ts:503-524
Data Enrichment
Beyond simply copying subgraph data, the event sync process enriches records by:
- Metadata Retrieval: Fetching full market details from decentralized storage (Irys/Arweave) using the
metadataHash. - Image Processing: Downloading market and event icons and persisting them to local object storage.
- Tag Normalization: Categorizing events into main categories (e.g., Politics, Sports, Crypto) based on predefined slugs.
- Sports Inference: Automatically identifying sports events and linking them to external providers (e.g., TheSportsDB or PandaScore) if confidence thresholds are met.
Sources: src/app/api/sync/events/route.ts:546-590, src/app/api/sync/events/route.ts:983-1010
Market Resolution Synchronization
Resolution sync monitors the "Resolution Subgraph" to detect when prediction markets have been finalized on-chain.
Processing Workflow
- Cursor Update: Identifies the last processed resolution timestamp.
- Condition Update: Updates the
conditionstable with the final status, resolution price, and resolution timestamp. - Payout Sync: Triggers
syncMissingOnChainResolvedPayoutsto record winning outcome distributions. - Cache Invalidation: Invalidates Next.js cache tags for the resolved events and the global event list to reflect the status change on the frontend.
Sources: src/app/api/sync/resolution/route.ts:133-210, src/app/api/sync/resolution/route.ts:400-450
Background Job Worker
The platform handles complex, multi-step on-chain operations (like deploying new markets) through a specialized job queue stored in the jobs table.
Event Creation Job Flow
The deploy_event_creation job processes deployments in a state-machine fashion:
| State | Description |
|---|---|
| Pending | Job is created and awaiting the next worker tick. |
| Processing | Worker has claimed the job and is communicating with the on-chain gateway. |
| Running | Transactions are being signed and submitted to the blockchain. |
| Completed | Market is successfully deployed and finalized. |
| Failed | Job exceeded max_attempts (default 6) after retries. |
Sources: src/app/api/sync/event-creations/route.ts:32-55, src/app/api/sync/event-creations/route.ts:384-410
Scheduling Infrastructure
The project supports two primary methods for triggering these background tasks.
1. PostgreSQL (pg_cron)
In Supabase-compatible environments, the scripts/migrate.ts script automatically configures native database crons using pg_net to issue HTTP GET requests to the API endpoints.
| Job Name | Schedule | Endpoint |
|---|---|---|
sync-events | Every 9 mins | /api/sync/events |
sync-resolution | Every 10 mins | /api/sync/resolution |
sync-volume | Every 5 mins | /api/sync/volume |
clean-jobs | Every hour | (Internal SQL) |
Sources: scripts/migrate.ts:285-350
2. Kubernetes CronJobs
For containerized deployments, the system defines standard Kubernetes CronJob resources that use curl to trigger the sync routes.
apiVersion: batch/v1
kind: CronJob
metadata:
name: kuest-sync-events
spec:
schedule: '2,11,20,29,38,47,56 * * * *'
jobTemplate:
spec:
template:
spec:
containers:
- name: trigger
image: curlimages/curl
command: ["sh", "-c", "curl -H 'Authorization: Bearer ${CRON_SECRET}' ${SITE_URL}/api/sync/events"]
Sources: infra/kubernetes/cronjobs.yaml:1-35
Conclusion
The data synchronization system provides a robust backbone for the prediction market, ensuring that on-chain events and resolutions are mirrored in the local database with minimal latency. By combining incremental subgraph polling with a stateful job worker, the system maintains data integrity while handling complex blockchain interactions in the background.