Dataset Management & Discovery

Relevant source files

The following files were used as context for generating this wiki page:

• apps/server/src/routes/v1/datasets.ts
• packages/db/src/schema/data.ts
• apps/server/src/agents/tools/search-db.ts
• apps/web/src/app/datasets/upload/page.tsx
• apps/workers/sync-worker/src/resource_embed.rs
• apps/server/src/agents/prompts/system.ts
• apps/data-worker/src/sources/ckan.rs

The Dataset Management & Discovery system in Poneglyph provides a centralized infrastructure for ingesting, indexing, and retrieving humanitarian data. It supports both manual user uploads and automated synchronization from external catalogs like CKAN. By combining traditional SQL filtering with advanced semantic search (pgvector), the system enables researchers and NGO workers to find actionable insights across diverse file formats including CSV, PDF, JSON, and Excel.

Sources: apps/server/src/agents/prompts/system.ts:1-20, packages/db/src/schema/data.ts:46-95

Data Architecture & Schema

The core of the system is a PostgreSQL database managed via Drizzle ORM. The schema is designed to handle multi-modal data, including metadata (titles, descriptions), physical file storage references (S3 keys), and high-dimensional vector embeddings for AI-powered search.

Entity Relationship

The relationship between sources, datasets, and tags is managed through relational tables and junction tables to ensure efficient filtering.

Core Data Models

Table	Field	Type	Description
datasets	id	uuid	Primary key, unique identifier.
datasets	embedding	vector(768)	Vector representation for semantic search.
datasets	s3_keys	text[]	Array of keys/URLs for file attachments.
datasets	dataset_status	enum	Lifecycle state: pending, approved, rejected, archived.
sources	source_type	enum	Origin of data: upload, external_url, or api.

Sources: packages/db/src/schema/data.ts:25-85, packages/db/src/migrations/meta/0004_snapshot.json

Dataset Ingestion & Processing

Datasets enter the system through two primary flows: manual user contributions via the web interface and automated synchronization workers.

Manual Upload Flow

Users can upload files (CSV, JSON, JSONL, Parquet) up to 50MB each. The frontend provides a real-time preview of how the dataset will appear in the discovery feed.

Automated Synchronization (CKAN)

The sync-worker periodically fetches packages from CKAN instances. It maps external metadata (notes, organization, resources) to the internal Poneglyph schema.

• Metadata Mapping: CKAN "notes" become the internal "description".
• Deduplication: Uses external_id (CKAN package ID) to prevent duplicate records.
• Resource Mapping: Normalizes file formats (e.g., xlsx/xls to "xlsx").

Sources: apps/data-worker/src/sources/ckan.rs:56-120

Discovery & Search Mechanisms

Discovery is implemented through a hybrid approach combining Full-Text Search (FTS) for keyword matching and Vector Similarity for semantic understanding.

Semantic Search (AI-Powered)

The system utilizes a searchDatabaseTool that converts user queries into embeddings and performs a cosine similarity search against the datasets table using the pgvector extension.

Filtering and Pagination

The REST API provides extensive filtering capabilities for the discovery grid, including:

• Status: Filters by approved/pending/archived states.
• File Type: Filters using the ANY(file_types) PostgreSQL operator.
• Tags: Performs complex intersections to ensure datasets match all requested tags.
• Sorting: Supports sorting by viewCount, downloadCount, or createdAt.

Sources: apps/server/src/routes/v1/datasets.ts:60-120

Vector Embedding Generation

To enable semantic discovery, the system generates embeddings for both metadata and resource content.

1. Metadata Embedding: A textual representation is built from the title, description, publisher, and tags.
2. Resource Embedding: The system attempts to extract text from supported file types (PDF, CSV, JSON, XLSX).
3. Averaging: If multiple resources are present, the metadata vector and file vectors are averaged to create a final 768-dimensional representative vector for the dataset.

pub async fn build_dataset_embedding(
    ds: &SourceDataset,
    metadata_text: &str,
    api_key: &str,
) -> Result<Vec<f32>> {
    let metadata_vector = embed::embed_text(metadata_text, api_key).await?;
    // ... logic to fetch and embed files ...
    Ok(average_vectors(all_vectors))
}

Sources: apps/workers/sync-worker/src/resource_embed.rs:20-50, apps/data-worker/src/sources/ckan.rs:135-160

API Endpoints for Discovery

The datasetsRouter (Hono) exposes the following endpoints for the frontend and AI agents:

Method	Endpoint	Description
GET	/api/v1/datasets	Paginated listing with FTS search and filters.
GET	/api/v1/datasets/:id	Full detail including source info and attachment metadata.
GET	/api/v1/datasets/:id/files/:index	Redirects to a presigned S3 URL or external source URL.

Sources: apps/server/src/routes/v1/datasets.ts:50-250

Summary

Dataset Management & Discovery in Poneglyph acts as a bridge between raw humanitarian data and analytical insights. By utilizing a hybrid storage model (PostgreSQL for metadata, S3 for files, and Vector embeddings for meaning), the system ensures that high-impact data is easily discoverable through both traditional UI navigation and AI-assisted research tools.