Cloud Storage & S3 Proxy
Relevant source files
The following files were used as context for generating this wiki page:
The Cloud Storage and S3 Proxy system provides a unified interface for managing binary assets within the Driveway Estimator Pro application. Its primary purpose is to handle the secure upload and retrieval of driveway photos, AI-generated material previews, and exported documents using Amazon S3 as the underlying storage provider.
This system leverages a proxy architecture where the application interacts with a "Forge" API to obtain presigned URLs. This allows the client or server to perform direct uploads to S3, reducing the load on the application server while maintaining secure access control over the assets.
Sources: README.md:37, server/storage.ts:1-6, todo.md:71
Storage Architecture
The storage implementation is designed around presigned URL patterns. Instead of passing large binary blobs through the application backend, the system requests temporary permission from the storage provider (S3) via the Forge API.
Key Components
- Forge Configuration: Validates and normalizes the required environment variables (
BUILT_IN_FORGE_API_URLandBUILT_IN_FORGE_API_KEY) to ensure the storage service can authenticate with the proxy provider. - Key Normalization: Ensures relative paths are properly formatted by stripping leading slashes and appending unique cryptographic hashes to prevent filename collisions.
- Presigned PUT Operations: Requests an S3 upload URL for a specific path, allowing the system to send data directly to storage.
- Presigned GET Operations: Generates temporary, secure links for retrieving private assets.
Sources: server/storage.ts:7-30, server/storage.ts:88-92
Data Flow for Uploads
The following diagram illustrates the flow of data when the application needs to store a new file, such as a driveway photo or a material preview.
The system uses a two-step process to ensure that sensitive storage credentials are never exposed to the client while still allowing high-performance direct-to-S3 uploads.
Sources: server/storage.ts:32-75
Integration with Database Schema
The application tracks storage assets by persisting both the absolute reference (Key) and the accessible URL in the projects table. This allows the system to manage file lifecycles and provides quick access for frontend rendering.
Storage-Related Fields
| Field | Type | Description |
|---|---|---|
photoUrl | text | The public-facing proxy path used by the frontend to display the original driveway photo. |
photoKey | text | The internal S3 identifier used for administrative operations and signed URL generation. |
previewImageUrl | text | The path to the AI-generated visualization of the selected asphalt/material. |
previewImageKey | text | The internal key for the AI-generated preview asset. |
Sources: drizzle/schema.ts:46-49, drizzle/schema.ts:58-59
Asset Security and Retrieval
Security is maintained by using a proxy path (/manus-storage/{key}) that results in a 307 Redirect to the actual S3 location. For restricted access, the system provides functions to generate signed URLs with expiration times.
Key Retrieval Functions
This sequence ensures that internal S3 paths and credentials remain protected behind the Forge API layer.
Sources: server/storage.ts:88-109
Practical Implementation Details
Unique Key Generation
To prevent overwriting existing files when multiple users upload assets with similar names (e.g., driveway.jpg), the system appends an 8-character hexadecimal hash suffix derived from a UUID.
function appendHashSuffix(relKey: string): string {
const hash = crypto.randomUUID().replace(/-/g, "").slice(0, 8);
const lastDot = relKey.lastIndexOf(".");
if (lastDot === -1) return `${relKey}_${hash}`;
return `${relKey.slice(0, lastDot)}_${hash}${relKey.slice(lastDot)}`;
}
Sources: server/storage.ts:24-29
Handling Buffers and Binary Data
The storage service is utilized by various modules, including the pdfExport service, which generates Buffer objects from jsPDF output. These buffers are then passed to storagePut to be persisted as project summaries.
Sources: server/services/pdfExport.ts:117-118, server/storage.ts:34
Summary
The Cloud Storage & S3 Proxy system acts as a secure, abstract layer for asset management in Driveway Estimator Pro. By using presigned URLs and a centralized storage service, the application ensures high performance, prevents filename collisions through cryptographic hashing, and maintains security via redirected proxy paths and temporary signed URLs. This architecture supports critical features such as driveway photo uploads, AI material previews, and PDF report generation.