Bulk Link Creation
Relevant source files
The following files were used as context for generating this wiki page:
- src/server/modules/bulk/services/BulkService.ts
- src/server/modules/bulk/BulkController.ts
- src/client/user/components/CreateUrlModal/CreateLinkForm.tsx
- src/server/repositories/UrlRepository.ts
- src/server/modules/user/services/UrlManagementService.ts
- src/client/user/actions/index.ts
- src/shared/constants.ts
- src/server/api/user/index.ts
Bulk Link Creation allows users to generate multiple shortened links simultaneously by uploading a CSV file containing original URLs. This feature automates the process of mapping long URLs to randomly generated short slugs, applying consistent tags across the batch, and optionally triggering background jobs for QR code generation.
The system enforces strict validation on the uploaded CSV to ensure link safety and system integrity, checking for blacklisted domains, circular redirects, and file size limits.
Architecture and Data Flow
The bulk creation process follows a multi-layered path from the React-based frontend to the PostgreSQL database, orchestrated by the BulkController and BulkService.
Execution Flow
- Frontend Input: The user selects the "From a .csv" tab in the
CreateLinkFormand uploads a file. - API Request: The client sends a
multipart/form-dataPOST request to/api/user/url/bulk. - Parsing & Validation: The
BulkServiceparses the CSV usingpapaparse, validating headers and link counts. - Mapping: The system generates unique random slugs for each valid long URL.
- Persistence: The
UrlRepositoryperforms a bulk database insertion within a single transaction. - Post-Processing: If enabled, the system initiates an asynchronous job to generate QR codes for the new links.
Sources: [src/client/user/components/CreateUrlModal/CreateLinkForm.tsx:313-356], [src/server/modules/bulk/BulkController.ts:46-77], [src/server/api/user/index.ts:107-122]
Process Sequence Diagram
This diagram illustrates the interaction between components during a bulk upload.
Sources: [src/server/modules/bulk/BulkController.ts], [src/server/modules/user/services/UrlManagementService.ts:182-194]
Validation Rules
The BulkService validates every row in the CSV file before any links are created. If any row fails validation, the entire process is aborted to maintain data consistency.
- Header Check: The first row must exactly match the constant
BULK_UPLOAD_HEADER("Original links to be shortened"). - Link Limit: The number of URLs must not exceed
BULK_UPLOAD_MAX_NUM(default is 1000). - Column Count: Each row must contain exactly one column.
- URL Safety: URLs must be valid, not blacklisted, and must not point back to the
ogHostname(circular redirect prevention). - Empty Rows: The service rejects files with empty rows or empty data strings.
Sources: [src/server/modules/bulk/services/BulkService.ts:25-99], [src/shared/constants.ts:4]
CSV Validation Logic Flow
Sources: [src/server/modules/bulk/services/BulkService.ts:48-99]
Component Specifications
API Endpoints
The following endpoint handles bulk link operations:
| Method | Path | Description |
|---|---|---|
POST | /api/user/url/bulk | Accepts a CSV file and optional tags to create multiple links. |
Sources: [src/server/api/user/index.ts:107-122]
Configuration Constants
Key constraints defined in the shared constants and server configuration:
| Constant | Value / Source | Description |
|---|---|---|
MAX_CSV_UPLOAD_SIZE | 5 MB | Maximum allowed file size for the CSV upload. |
BULK_UPLOAD_HEADER | "Original links to be shortened" | Required text for the first row of the CSV. |
BULK_UPLOAD_MAX_NUM | Defaults to 1000 | Maximum number of links in a single batch. |
BULK_UPLOAD_RANDOM_STR_LENGTH | Defaults to 8 | Length of the generated shortUrl slugs. |
Sources: [src/shared/constants.ts:1-4], [README.md:129-130]
Database Implementation
The UrlRepository.bulkCreate method manages the persistence of link mappings and their associated tags. It uses a Sequelize transaction to ensure that all URLs and their tag associations are created successfully or not at all.
// Simplified logic from UrlRepository.ts
const bulkUrlObjects = urlMappings.map(({ shortUrl, longUrl }) => ({
shortUrl,
longUrl,
userId,
source: StorableUrlSource.Bulk,
tagStrings: tags ? tags.join(TAG_SEPARATOR) : '',
safeBrowsingExpiry: getSafeBrowsingExpiryDate({ longUrl }).toISOString(),
}))
const urls = await Url.bulkCreate(bulkUrlObjects, { transaction: t })
Sources: [src/server/repositories/UrlRepository.ts:577-611]
User Interface
The CreateLinkForm component provides the interface for bulk uploads. It includes:
- Template Download: A button to download a sample CSV file to ensure users follow the correct format.
- File Input: A
FileInputFieldrestricted to.csvfiles. - Tagging: A
TagsAutocompletecomponent to apply up to 3 tags to all links in the batch. - Feedback: Displays file name, size, and specific validation errors (e.g., "Only CSV format files are allowed").
Sources: [src/client/user/components/CreateUrlModal/CreateLinkForm.tsx:313-356], [src/client/user/actions/index.ts:1004-1025]
Conclusion
Bulk Link Creation in GoGovSG provides a robust mechanism for high-volume link management. By combining frontend validation, server-side CSV parsing, and transactional database operations, the system ensures that large batches of links are created safely and efficiently. The integration with background jobs further enhances the feature by providing automated QR code generation for the newly created link sets.