CKAN Data Synchronization
Relevant source files
The following files were used as context for generating this wiki page:
The CKAN Data Synchronization system is a core component of the Poneglyph project, responsible for harvesting humanitarian datasets from external CKAN-based platforms. Its primary purpose is to automate the ingestion of metadata and resource content, transforming them into a searchable, AI-enhanced repository for NGO workers and researchers.
This system operates as a background worker (sync-worker) that interfaces with the CKAN Action API, executes deduplication logic against a PostgreSQL database, and generates vector embeddings for both metadata and supported file resources to enable semantic search capabilities.
Sources: apps/workers/sync-worker/README.md, apps/workers/sync-worker/src/sync.rs:11-30
Architecture and Data Flow
The synchronization process follows a structured pipeline: fetching, deduplication, content processing (embedding), and database persistence.
High-Level Sync Flow
The following diagram illustrates the sequence of operations performed during a synchronization task.
CKAN Source Integration
The system interacts with the CKAN package_search API endpoint. It uses a paginated approach to fetch all datasets matching a specific query, processing them in batches of 100.
CKAN Data Mapping
External CKAN packages are mapped to the internal SourceDataset structure. The mapping logic handles the conversion of CKAN-specific fields to the Poneglyph schema.
| CKAN Field | Internal Field | Description |
|---|---|---|
id | external_id | Unique identifier from the source for deduplication. |
title | title | The name of the dataset. |
notes | description | Detailed description or summary. |
metadata_created | publication_date | Parsed ISO 8601 timestamp. |
organization.title | publisher | The publishing entity. |
resources | resource_urls | List of direct download links for files. |
Sources: apps/workers/sync-worker/src/sources/ckan.rs:11-41, apps/data-worker/src/sources/ckan.rs:88-124
File Type Normalization
The system normalizes various resource formats into a standard set of supported types.
fn map_file_type(fmt: &str) -> String {
match fmt {
"pdf" => "pdf".to_string(),
"csv" => "csv".to_string(),
"xlsx" | "xls" => "xlsx".to_string(),
"json" => "json".to_string(),
"docx" => "docx".to_string(),
_ => "other".to_string(),
}
}
Sources: apps/workers/sync-worker/src/sources/ckan.rs:126-134
AI-Powered Embedding Generation
A critical feature of the synchronization process is the generation of semantic embeddings. This allows the system to understand the context of datasets beyond simple keyword matching.
Metadata Text Construction
Before embedding, a text representation is built from the dataset's metadata, including the title, organization, tags, and a summary of available resources. This text is truncated at 6000 characters to fit model constraints.
Sources: apps/data-worker/src/sources/ckan.rs:136-184
Resource Processing
If a dataset contains supported files (PDF, CSV, JSON, XLSX), the worker attempts to fetch and embed the actual content of those files.
- PDFs: Uploaded to the Gemini File API for processing.
- Plain Text (CSV/JSON): Content is downloaded and truncated for local processing.
- Excel (XLSX/XLS): Text is extracted from all worksheets using the
calaminelibrary.
The final dataset embedding is an average of the metadata vector and all individual resource vectors.
Sources: apps/workers/sync-worker/src/resource_embed.rs:20-65, apps/workers/sync-worker/src/resource_embed.rs:172-205
Database Schema and Persistence
The system uses a PostgreSQL database managed via Drizzle ORM. The datasets table is the primary storage for synchronized records.
Dataset Data Model
The schema includes a vector type for storing embeddings and supports HNSW indexing for efficient similarity searches.
| Field | Type | Constraints |
|---|---|---|
id | uuid | Primary Key, Default Random |
external_id | text | Unique (Source package ID) |
source_id | uuid | Foreign Key -> sources.id |
embedding | vector(768) | Used for semantic search |
dataset_status | enum | pending, approved, rejected, archived |
Sources: packages/db/src/schema/data.ts:54-102
Sync Logging
Every execution is recorded in the sync_logs table to track progress and errors.
Summary
The CKAN Data Synchronization system provides a robust pipeline for humanitarian data ingestion. By combining traditional API harvesting with modern AI embedding techniques and structured logging, it ensures that Poneglyph maintains an up-to-date and contextually searchable repository of global survey data. Significant emphasis is placed on deduplication via external_id and the best-effort embedding of resource content to maximize data utility.