Core Data Worker Application
Relevant source files
The following files were used as context for generating this wiki page:
The Core Data Worker Application (also referred to as the sync-worker) is a critical background service within the Poneglyph ecosystem responsible for the ingestion, synchronization, and enrichment of humanitarian datasets from external and internal sources. Its primary role is to fetch data from providers like CKAN, process associated metadata, and generate vector embeddings to support AI-powered search and analysis capabilities across the platform.
The application is designed to be highly portable, supporting various runtime environments including local execution, AWS Lambda, and Google Cloud Run. It ensures that the Poneglyph database remains up-to-date by monitoring external repositories and writing synchronized records into a PostgreSQL database managed by Drizzle ORM.
Sources: apps/workers/sync-worker/README.md:1-15, packages/db/src/migrations/meta/0004_snapshot.json
Data Ingestion and Synchronization
The worker implements specialized logic to interface with the CKAN API, a common platform for open data. It performs paginated searches to retrieve dataset packages, which are then mapped into the internal SourceDataset structure.
CKAN Ingestion Flow
The ingestion process follows a loop that queries the CKAN package_search endpoint, handling pagination through offsets until all relevant datasets are retrieved.
The mapping process extracts critical metadata including unique identifiers, titles, descriptions (notes), and organization details. It also normalizes file formats to ensure compatibility with downstream embedding processes.
Sources: apps/workers/sync-worker/src/sources/ckan.rs:50-104, apps/data-worker/src/sources/ckan.rs:50-104
Data Mapping Specifications
The following table describes how external CKAN fields are mapped to the internal SourceDataset model:
| Source Field (CKAN) | Internal Field | Description |
|---|---|---|
id | external_id | Unique identifier from the source provider. |
title | title | The name of the dataset. |
notes | description | Detailed text describing the dataset contents. |
organization.title | publisher | The name of the entity that published the data. |
metadata_created | publication_date | Date the record was first created at the source. |
resources | resource_urls | A collection of links to actual data files. |
Sources: apps/workers/sync-worker/src/sources/ckan.rs:106-146, apps/data-worker/src/sources/ckan.rs:106-146
Vector Embedding System
A core feature of the worker is the generation of vector embeddings. This allows the Poneglyph Research Assistant to perform semantic searches over datasets. The system employs a dual-embedding strategy: embedding the textual metadata and embedding the content of the attached data files.
Embedding Logic and Best Practices
- Metadata Embedding: Textual summaries are constructed from the title, description, publisher, and tags. This text is truncated at approximately 6,000 characters to fit model constraints.
- Resource Embedding: The worker attempts to download and embed the actual content of files (PDF, CSV, JSON, XLSX, XLS).
- Vector Averaging: To create a single representation of a dataset, the system averages the metadata vector with any successfully generated resource vectors.
Resource Processing Constraints
To prevent unbounded resource usage during synchronization, the worker implements strict limits on file processing:
| Constraint | Value / Limit | Source File |
|---|---|---|
| Max Concurrent Fetches | Configurable (limit) | resource_embed.rs:32 |
| Supported Formats | PDF, CSV, JSON, XLSX, XLS | resource_embed.rs:60-65 |
| Max Text Truncation | max_text_chars | resource_embed.rs:271 |
| Embedding Dimension | 768 (pgvector) | 0004_snapshot.json |
Sources: apps/workers/sync-worker/src/resource_embed.rs:29-70, packages/db/src/migrations/meta/0004_snapshot.json
Database Schema and Synchronization Logging
The worker interacts with several tables in the PostgreSQL database to track the status of datasets and the results of synchronization jobs.
Key Database Tables
datasets: The primary table for dataset records. It includes anembeddingcolumn of typevector(768)with an HNSW index for fast similarity searches.sync_logs: Tracks the history of synchronization attempts, including counts of items added, updated, or archived, and any error messages encountered.dataset_tags/tags: Handles the many-to-many relationship between datasets and searchable tags.
Runtime and Deployment
The application is designed for flexibility in deployment. The source code contains conditional branches to support different execution environments:
- Cloud Run / Local: A standard TCP/HTTP server path (using Hono or internal Rust handlers) triggered by POST requests to
/sync. - AWS Lambda: Specialized handlers compatible with
Dockerfile.lambdafor event-driven execution via CloudWatch or EventBridge.
Sources: apps/workers/sync-worker/README.md:5-15
Conclusion
The Core Data Worker is the backbone of data intelligence in Poneglyph. By automating the extraction of humanitarian data from CKAN and enriching it through multi-modal vector embeddings, it transforms raw data into a structured, searchable knowledge base for researchers and NGO workers. Its robust logging and flexible deployment options ensure it can scale according to the volume of incoming datasets.