Upload Worker Microservice
Relevant source files
The following files were used as context for generating this wiki page:
The Upload Worker Microservice is a Rust-based background processing unit responsible for the ingestion, transformation, and embedding of datasets uploaded by users. It serves as the asynchronous processing engine that handles heavy computational tasks such as text extraction from various file formats and generating high-dimensional vector embeddings to support semantic search within the Poneglyph ecosystem.
This microservice acts as the second stage of the upload pipeline. After the API Server receives files and metadata, it offloads the processing to this worker via a message queue (Pub/Sub or RabbitMQ). The worker ensures that humanitarian data—ranging from PDFs and CSVs to Excel sheets—is structured and indexed properly in the PostgreSQL database using vector embeddings generated via the Gemini AI API.
Sources: apps/workers/upload-worker/src/upload.rs, apps/server/src/routes/v1/upload.ts:1-133, apps/data-worker/src/upload.rs:1-447
System Architecture and Data Flow
The Upload Worker follows a producer-consumer pattern. The primary API server validates the initial request and uploads raw files to S3/R2 storage before dispatching a message to the worker.
Integration Flow
The following sequence diagram illustrates the lifecycle of a dataset from user submission to indexed availability.
The worker utilizes reqwest for HTTP communication with the Gemini File API and R2 storage, and sqlx for database interactions.
Sources: apps/server/src/routes/v1/upload.ts:51-110, apps/workers/upload-worker/src/upload.rs:50-120, apps/data-worker/src/upload.rs:77-179
Core Data Structures
The system relies on standardized schemas for message passing and callback notifications, ensuring consistency between the TypeScript-based server and the Rust-based worker.
Upload Message Schema
| Field | Type | Description |
|---|---|---|
upload_id | String | Unique UUID for the upload session. |
user_id | String | ID of the uploading user (or "anonymous"). |
title | String | Human-readable title of the dataset. |
description | String | Detailed description for context. |
attachments | Vec<AttachmentInfo> | List of S3 keys and presigned URLs for files. |
callback_url | String | URL to notify upon completion. |
Sources: apps/workers/upload-worker/src/upload.rs:13-33, packages/schemas/src/upload.ts:33-52
Document Processing and Embedding Logic
The worker implements a multi-strategy embedding approach. It first embeds the dataset metadata (title, description, summary) and then attempts to embed individual attachments based on their MIME types.
File Format Support
The worker provides specialized handling for various formats:
- Plain Text/CSV/JSON: Content is read, lossily converted to UTF-8, and truncated (6000 character limit) for the Gemini API.
- PDF: Files are uploaded to the Gemini File API via a resumable upload protocol for direct processing.
- Excel (XLSX/XLS): The
calaminelibrary is used to extract text from all workbook sheets.
Vector Aggregation
Once multiple vectors are obtained (one for metadata and one per supported attachment), the worker performs an element-wise average to produce a single final embedding representing the entire dataset.
Resumable Upload and Polling
For large files processed via the Gemini File API, the worker implements a three-step resumable upload protocol:
- Initiation: A POST request to the Gemini upload endpoint to get a unique upload URL.
- Transmission: A PUT request to stream the raw bytes.
- Polling: If the file is not immediately
ACTIVE, the worker polls the API at 1.5-second intervals (up to 20 attempts).
Sources: apps/workers/upload-worker/src/upload.rs:218-305, apps/data-worker/src/upload.rs:334-406
Infrastructure Configuration
The microservice is designed for containerized deployment, typically via Cloud Run. Infrastructure variables define scaling and connectivity parameters.
| Variable | Default Value | Description |
|---|---|---|
worker_service_name | poneglyph-upload-worker | Cloud Run service name. |
worker_max_scale | 1 | Maximum instances for processing. |
pubsub_upload_topic | poneglyph-upload | Topic for job triggers. |
worker_container_port | 8080 | Port for the HTTP callback listener. |
Sources: terraform/variables.tf:19-65
Conclusion
The Upload Worker Microservice is critical for Poneglyph's semantic capabilities. By offloading document parsing and vector generation to an asynchronous Rust environment, the system maintains high responsiveness while building a rich, searchable knowledge base of humanitarian data. Its robust handling of diverse file formats and sophisticated embedding aggregation ensures that complex datasets are accurately represented in the project's vector space.