S3 Storage Integration
Relevant source files
The following files were used as context for generating this wiki page:
The S3 Storage Integration within Poneglyph serves as the primary persistence layer for large-scale humanitarian datasets, field reports, and associated metadata. It provides a robust interface for handling multi-part file uploads, generating presigned URLs for secure access, and managing storage for both primary data assets and UI-related media like thumbnails.
This system is integrated into the Poneglyph Research Assistant ecosystem, allowing AI agents to search and retrieve curated field reports uploaded by NGOs. The architecture relies on S3-compatible storage (such as Cloudflare R2) to store documents including PDFs, CSVs, and JSON files, which are later processed for vector embeddings and search.
Sources: apps/server/src/routes/v1/upload.ts:36-39, apps/server/src/agents/prompts/system.ts:10-15
Architecture and Data Flow
The storage integration follows a multi-stage pipeline: ingestion via a Hono-based API, parallel upload to S3-compatible storage, and asynchronous processing via a background worker.
Upload Pipeline
When a user submits a dataset via the web interface, the server receives a multipart/form-data request. The server generates a unique upload_id (UUID) to partition files within the bucket. Files and optional thumbnails are uploaded in parallel to the S3 endpoint.
The diagram shows the lifecycle of a file from initial client upload to eventual processing by the sync worker.
Sources: apps/server/src/routes/v1/upload.ts:47-92, terraform/variables.tf:65-75
File Handling and Validation
The system enforces strict validation on file types and sizes to ensure data integrity and prevent storage abuse.
Supported File Types
The integration maps MIME types to specific file extensions for consistent S3 key generation and database categorization.
| Category | MIME Type | Extension |
|---|---|---|
| Document | application/pdf | |
| Document | text/csv | csv |
| Document | application/json | json |
| Spreadsheet | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | xlsx |
| Spreadsheet | application/vnd.ms-excel | xls |
| Image | image/jpeg, image/png, image/webp | jpg, png, webp |
Sources: apps/server/src/routes/v1/upload.ts:18-30, packages/schemas/src/upload.ts:3-13
Validation Constraints
- Maximum File Size: 50MB per file.
- Requirement: At least one document file is required for a dataset upload.
- Thumbnail: Optional image file (JPG, PNG, WebP).
Sources: packages/schemas/src/upload.ts:15-50
Storage Configuration
The integration is configured via environment variables and Terraform, supporting S3-compatible endpoints.
S3 Configuration Parameters
| Variable | Description | Sensitivity |
|---|---|---|
s3_access_key | IAM/R2 access key ID | Sensitive |
s3_secret_key | IAM/R2 secret access key | Sensitive |
s3_bucket_name | Target bucket for uploads | Non-sensitive |
s3_endpoint | Custom S3 endpoint (e.g., Cloudflare R2) | Non-sensitive |
s3_region | AWS region or 'auto' for R2 | Non-sensitive |
Sources: terraform/variables.tf:101-125
Data Modeling and Persistence
S3 keys are stored in the PostgreSQL database using Drizzle ORM to link physical storage with application metadata.
Dataset Schema Integration
The datasets table stores arrays of S3 keys and file types to maintain a reference to the uploaded objects.
export const datasets = pgTable("datasets", {
id: uuid("id").primaryKey().defaultRandom(),
title: varchar("title", { length: 500 }).notNull(),
thumbnailS3Key: text("thumbnail_s3_key"),
s3Keys: text("s3_keys").array(), // References to S3 objects
fileTypes: fileTypeEnum("file_types").array(),
// ... other fields
});
Sources: packages/db/src/schema/data.ts:60-100
Implementation Details
The server utilizes the @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner packages to interact with the storage layer.
S3 Key Structure
Keys are organized hierarchically to prevent collisions and allow for easy cleanup:
- Datasets:
uploads/${uploadId}/${uuid}.${ext} - Thumbnails:
uploads/${uploadId}/thumbnail.${ext}
Sources: apps/server/src/routes/v1/upload.ts:66-79, apps/server/package.json:28-29
Parallel Processing
The server uses Promise.all to execute S3 uploads for multiple files concurrently, minimizing the time the Hono request remains active before offloading to the message queue.
The sequence diagram illustrates the parallel nature of the file upload process within the server route.
Sources: apps/server/src/routes/v1/upload.ts:64-92
Summary
The S3 Storage Integration provides Poneglyph with a scalable way to handle large humanitarian datasets. By combining strict Zod-based validation, parallel S3 uploads, and an asynchronous processing model via Pub/Sub, the system ensures that high-volume data ingestion does not block user interactions while maintaining a structured, searchable repository of field reports.