Data Uploads & File Processing
Relevant source files
The following files were used as context for generating this wiki page:
The Data Uploads & File Processing system is a core component of the Poneglyph platform, enabling researchers and NGOs to contribute humanitarian datasets to a centralized repository. The system handles multi-part form data, manages file storage in S3-compatible buckets (R2), and orchestrates asynchronous background processing to generate AI-powered embeddings for search and discovery. Sources: apps/server/src/routes/v1/upload.ts, apps/web/src/app/datasets/upload/page.tsx, apps/server/src/agents/prompts/system.ts:8-13
This architecture ensures that large files (up to 50MB per file) do not block the main API thread. By utilizing a worker-based model, Poneglyph can perform intensive tasks such as text extraction from PDFs and Excel files, vector embedding generation via the Gemini API, and automated tagging without impacting user experience. Sources: packages/schemas/src/upload.ts:13, apps/workers/upload-worker/src/upload.rs:107-135
System Architecture & Data Flow
The upload process follows a "queued" pattern where the API server acts as a gateway, validating requests and offloading the heavy lifting to specialized Rust-based workers.
Diagram 1: The end-to-end data flow from user submission to database persistence and AI indexing.
Sources: apps/server/src/routes/v1/upload.ts:46-107, apps/data-worker/src/upload.rs:55-147
Upload Lifecycle Stages
- Submission: Users provide metadata (title, description, tags) and files via the frontend. Sources: apps/web/src/app/datasets/upload/page.tsx:210-340
- Staging: The API server receives the files, generates unique UUIDs for the upload session, and stores the raw buffers in R2. Sources: apps/server/src/routes/v1/upload.ts:60-91
- Queuing: A message containing S3 keys and metadata is published to the processing queue. Sources: apps/server/src/routes/v1/upload.ts:98-109
- Processing: The Rust worker downloads the files, extracts content based on MIME type, and creates a unified vector embedding. Sources: apps/workers/upload-worker/src/upload.rs:46-95
- Completion: The worker triggers a callback to the API server to update the status and notify the user. Sources: apps/workers/upload-worker/src/upload.rs:245-283
API Endpoints & Validation
The server implements a strict validation layer using Zod to ensure data integrity before any file processing occurs.
Endpoint Specifications
| Method | Path | Description | Auth Required |
|---|---|---|---|
POST | /api/upload | Primary upload endpoint for datasets and thumbnails. | Yes |
POST | /api/upload/callback | Internal webhook for workers to report processing results. | Secret Header |
Sources: apps/server/src/routes/v1/upload.ts:46, 117
Request Schema (UploadRequestSchema)
The system enforces constraints on file sizes and types to maintain performance and compatibility.
- Max File Size: 50MB per file.
- Supported Documents: PDF, CSV, JSON, XLSX, XLS, DOCX.
- Supported Images: JPEG, JPG, PNG, WEBP.
- Metadata: Title (required), Description (required), Summary (optional), Publisher (optional), Tags (optional, comma-separated).
Sources: packages/schemas/src/upload.ts:3-50
Background Processing & AI Embedding
The Rust workers are responsible for transforming raw files into searchable intelligence. This is achieved by extracting text and using the Gemini API to create a 768-dimensional vector representing the dataset. Sources: apps/data-worker/src/upload.rs:200-210
Content Extraction Logic
The worker uses different strategies based on the file's MIME type:
- PDF: Uploaded to Gemini's File API for native document embedding. Sources: apps/workers/upload-worker/src/upload.rs:112-117
- Text (CSV/JSON): Content is read directly and truncated to 6,000 characters to fit model context windows. Sources: apps/workers/upload-worker/src/upload.rs:118-125
- Excel (XLSX/XLS): Parsed using the
calaminecrate to extract all cell values into a single text block. Sources: apps/workers/upload-worker/src/upload.rs:213-241
Vector Averaging
To represent a multi-file dataset as a single entry in the vector database, the system calculates the element-wise average of all generated vectors (metadata vector + individual file vectors).
Diagram 2: The process of merging metadata and file-level embeddings into a single dataset representation.
Sources: apps/workers/upload-worker/src/upload.rs:90, 243-255, apps/workers/sync-worker/src/resource_embed.rs:215-229
Error Handling & Reliability
The system is designed with fault tolerance for both network and API failures.
- Retry Logic: Messages that fail processing are republished to the queue with an incremented
_retrycount. If failures persist beyond 2 attempts, the message is moved to afailed_queuefor manual inspection. Sources: apps/data-worker/src/upload.rs:112-145 - Gemini Polling: For large files, the worker polls the Gemini File API until the file state becomes
ACTIVEbefore requesting embeddings. Sources: apps/workers/upload-worker/src/upload.rs:175-211 - Callback Security: The
/api/upload/callbackendpoint is protected by a shared secret (UPLOAD_CALLBACK_SECRET) to prevent unauthorized status updates. Sources: apps/server/src/routes/v1/upload.ts:118-124
Summary
The Data Uploads & File Processing system provides a robust pipeline for ingesting heterogeneous humanitarian data. By combining a Next.js frontend, a Hono API gateway, and high-performance Rust workers, Poneglyph ensures that datasets are not just stored, but transformed into actionable, AI-searchable assets. The use of S3 for storage and RabbitMQ for orchestration allows the platform to scale horizontally to meet the needs of researchers worldwide. Sources: README.md:3-15, apps/web/src/app/datasets/upload/page.tsx:143-158