File Uploads & S3 Management
Relevant source files
The following files were used as context for generating this wiki page:
GoGovSG provides a robust file upload system that allows authorized users to shorten files instead of standard URLs. The system manages the end-to-end lifecycle of a file, from client-side selection and validation to server-side threat scanning and permanent storage in Amazon S3.
The core of this system involves a multipart/form-data API that integrates with S3 via the AWS SDK. Files are stored as public objects when active, providing a direct download link through the GoGovSG domain. The system also supports bulk uploads via CSV files, which triggers serverless processes for QR code generation and data persistence.
Sources: src/server/api/user/index.ts, README.md:120-125
Architecture and Data Flow
The file upload process begins at the React frontend where users select a file. The application validates the file size and type before sending it to the Express backend. The backend uses express-fileupload middleware to handle the stream, performs virus checks, and finally invokes the UrlRepository to persist metadata in the database and the binary in S3.
File Upload Sequence
The following diagram illustrates the sequence of actions when a user uploads a file to create a short link.
Client-Side Implementation
The client uses a specialized FileInputField component to handle user interactions. This component provides feedback on file selection, size, and errors. The CreateLinkForm manages three upload modes: URL, File, and Bulk.
Validation Constraints
The system enforces strict limits to ensure performance and security:
| Feature | Constraint | Source |
|---|---|---|
| Standard File Upload | Max 20MB | src/server/api/user/index.ts:50 |
| Bulk CSV Upload | Max 5MB | src/server/api/user/index.ts:57 |
| Allowed Types | .csv (for bulk) | src/client/user/components/CreateUrlModal/CreateLinkForm.tsx:340 |
| Character Set | lowercase letters, numbers, hyphens | src/client/user/components/CreateUrlModal/CreateLinkForm.tsx:288 |
Sources: src/client/user/components/CreateUrlModal/CreateLinkForm.tsx:140-160, src/server/api/user/index.ts:50-60
Server-Side Management
The server manages S3 interaction through the S3ServerSide class (referenced in tests as S3Interface) and the UrlRepository.
S3 Service Operations
The S3ServerSide class encapsulates the AWS SDK commands:
- uploadFileToS3: Sends a
PutObjectCommandwithACL: public-readandCacheControl: no-cache. - setS3ObjectACL: Modifies the visibility of an existing file (e.g., switching between
PublicandPrivatewhen a link is activated or deactivated). - buildFileLongUrl: Constructs the S3 URL using the bucket name and file key.
Sources: test/server/services/aws.test.ts:40-70, test/server/repositories/UrlRepository.test.ts:160-200
State Management Logic
When a file link's state changes, the system synchronizes the S3 Object Access Control List (ACL) to prevent unauthorized access to inactive files.
Bulk Uploads and S3 Integration
The bulk upload feature processes CSV files containing multiple links. Upon successful validation, the system triggers a job that generates QR codes in various formats (SVG/PNG) and uploads them to S3 in ZIP archives.
Bulk Generation Flow
- Upload: User uploads a CSV (max 1000 links).
- Processing: The
bulk-qrcode-generationLambda reads the mappings. - Storage:
- Generates a CSV of short links and uploads to
${jobItemId}/generated.csv. - Generates SVG QR codes, zips them, and uploads to
${jobItemId}/generated_svg.zip. - Generates PNG QR codes, zips them, and uploads to
${jobItemId}/generated_png.zip.
- Generates a CSV of short links and uploads to
Sources: src/server/serverless/bulk-qrcode-generation/index.js:15-50, README.md:120-130
Configuration
S3 management relies on environment variables defined during infrastructure setup.
| Variable | Description | Required |
|---|---|---|
AWS_S3_BUCKET | The name of the bucket for storing uploads. | Yes |
AWS_ACCESS_KEY_ID | Credential for modifying files on S3. | Yes |
AWS_SECRET_ACCESS_KEY | Secret for S3 operations. | Yes |
BULK_GENERATION_BUCKET | S3 bucket specifically for bulk QR outputs. | Yes (Lambda) |
Sources: README.md:120-125, README.md:215-225
Conclusion
File Uploads and S3 Management in GoGovSG integrate client-side validation with secure server-side storage. By leveraging S3 ACLs synchronized with database states, the system ensures that files are only accessible when their corresponding short links are active. The addition of serverless bulk generation allows for scalable processing of large link datasets and their associated visual assets.