Vector Embeddings & Search
Relevant source files
The following files were used as context for generating this wiki page:
The Poneglyph platform utilizes vector embeddings to enable semantic search capabilities across its repository of humanitarian datasets, field reports, and NGO documents. By converting unstructured metadata and file content into high-dimensional numerical vectors, the system allows AI agents and researchers to find relevant information based on conceptual meaning rather than simple keyword matching.
The architecture consists of a multi-stage pipeline: data ingestion from external sources (like CKAN) or user uploads, content extraction from various file types (PDF, CSV, Excel), embedding generation via the Gemini API, and vector similarity search using PostgreSQL with the pgvector extension.
Data Ingestion & Pre-processing
Before a dataset can be searched, the system must extract and normalize text to form an "embedding string." This process occurs in both the sync-worker (for external sources) and the upload-worker (for user contributions).
Text Construction Logic
The system builds a comprehensive text representation of a dataset by concatenating specific metadata fields. This text is typically truncated to approximately 6,000 characters to fit model constraints.
| Component | Source Fields | Extraction Logic |
|---|---|---|
| Metadata | Title, Description, Publisher | Concatenated with labels (e.g., "Organization: {name}") |
| Tags | Dataset Tags | Joined as a comma-separated list |
| Resources | Filenames, Formats | Formatted as "[format] filename" |
Sources: apps/data-worker/src/sources/ckan.rs:114-152, apps/workers/upload-worker/src/upload.rs:109-119
File Content Extraction
To improve search accuracy, Poneglyph embeds the actual content of attached files alongside metadata.
- PDFs: Uploaded to the Gemini File API for specialized processing.
- Plain Text (CSV/JSON/JSONL): Read directly and truncated.
- Excel (XLSX/XLS): Parsed using the
calaminelibrary to extract cell values from all worksheets.
Sources: apps/workers/sync-worker/src/resource_embed.rs:72-101, apps/workers/upload-worker/src/upload.rs:252-277
Embedding Generation Pipeline
Poneglyph employs a "Vector Averaging" strategy. Instead of generating a single embedding for an entire dataset, it generates individual vectors for the metadata and each supported resource file, then calculates the centroid (average) to represent the dataset in vector space.
Embedding Data Flow
The following diagram illustrates the lifecycle of a dataset from ingestion to its final vector representation in the database.
The system averages the metadata vector with up to a configured limit of resource vectors (defined in max_resources_per_dataset).
Sources: apps/workers/sync-worker/src/resource_embed.rs:23-52, apps/workers/upload-worker/src/upload.rs:121-140
Semantic Search & Agent Tools
The searchDatabaseTool provides the primary interface for AI agents to query the vector store. It utilizes pgvector for performing cosine similarity searches.
The Search Process
- Query Embedding: The user's natural language query is converted into a vector using the same embedding model.
- Cosine Similarity: The system calculates the distance between the query vector and all stored dataset embeddings.
- Result Enrichment: Top results are fetched along with metadata, tags, and presigned S3 URLs for file access.
Implementation Detail: SQL Raw Execution
The search tool uses raw SQL instead of the Drizzle ORM query builder to avoid type conflicts between the ai package and the project's internal database client. The similarity score is calculated as 1 - (d.embedding <=> ${embeddingStr}::vector).
Sources: apps/server/src/agents/tools/search-db.ts:18-22, apps/server/src/agents/tools/search-db.ts:54-58
Configuration & Constraints
| Configuration Key | Purpose | Default/Context |
|---|---|---|
TOP_K | Number of search results returned to the agent | 5 |
max_text_chars | Max characters sent for embedding | 6,000 |
max_resources_per_dataset | Limit on files included in average vector | Defined in worker config |
embedding Column | Database column for vector storage | vector type in PostgreSQL |
Sources: apps/server/src/agents/tools/search-db.ts:11, apps/workers/sync-worker/src/resource_embed.rs:311, apps/data-worker/src/sources/ckan.rs:141-152
Conclusion
The Vector Embedding and Search system provides the foundation for Poneglyph's "Humanitarian Intelligence." By processing both metadata and deep file contents into a unified vector representation, the platform ensures that critical field data remains discoverable even when queries do not match exact keywords, enabling agents to synthesize insights across disparate datasets effectively.