Media Upload & Processing

Relevant source files

The following files were used as context for generating this wiki page:

• apps/api/src/module/video/video.service.ts
• apps/portal/app/services/api/image.ts
• apps/portal/app/services/api/video.ts
• apps/extensions/src/app/services/api/image.ts
• apps/extensions/src/app/services/api/video.ts
• apps/extensions/src/content/panel/hooks/useManageVideoData.ts
• apps/api/src/module/gauzy/services/gauzy-upload.service.ts
• apps/extensions/src/app/services/screenshots.ts
• apps/extensions/src/content/panel/screens/uploadScreen/UploadScreen.tsx

The Media Upload & Processing system is a core component of the Ever Rec platform, responsible for handling the ingestion, processing, storage, and management of user-generated images and videos. This system supports uploads from various clients, including the web portal and the browser extension, ensuring that captured media is securely stored and its metadata is recorded in the database.

The architecture is designed to handle different media formats and upload methods, such as direct file uploads, base64-encoded image data, and video blobs generated from screen recordings. On the backend, it leverages Firebase for storage and database operations, with additional processing steps like video correction using FFMPEG.

Architecture Overview

The system consists of frontend clients (Portal, Extension), a NestJS-based API backend, and Firebase services for persistence. Clients capture or select media and send it to the API, which then processes the media, stores the file in Firebase Storage, and saves associated metadata to the Firebase Realtime Database.

This diagram illustrates the high-level data flow for media uploads.

API Endpoints

The API exposes several endpoints for handling image and video uploads. These endpoints support different payload types, including multipart form data for files and JSON for base64-encoded data.

Image Endpoints

Method	Endpoint	Description	Payload Type
POST	/api/v1/image/upload	Uploads a base64-encoded image. Used by the extension after a screen capture.	application/json
POST	/api/v1/image/upload/file	Uploads an image file from the user's local drive.	multipart/form-data
PUT	/api/v1/image/single	Updates an existing image file (e.g., after editing) by providing a new blob.	multipart/form-data
PUT	/api/v1/image/single-data	Updates the metadata of an image, such as title, parent folder, trash status, likes, views, and markers.	application/json
POST	/api/v1/image/original	Saves a reference to an original image.	application/json
POST	/api/v1/image/update-original	Updates the original image file with new data.	application/json

Sources: apps/portal/app/services/api/image.ts:10-20, 22-34, apps/extensions/src/app/services/api/image.ts:7-18, 20-32

Video Endpoints

Method	Endpoint	Description	Payload Type
POST	/api/v1/video/upload	Uploads a video blob, typically from a browser recording.	multipart/form-data
POST	/api/v1/video/upload/file	Uploads a video file from the user's local drive.	multipart/form-data
PUT	/api/v1/video/single	Updates an existing video file (e.g., after trimming) by providing a new blob.	multipart/form-data
PUT	/api/v1/video/single-data	Updates the metadata of a video, such as title, parent folder, and trash status.	application/json
POST	/api/v1/video/setPosterRef	Sets a custom poster image for a video.	multipart/form-data

Sources: apps/portal/app/services/api/video.ts:15-25, 27-39, apps/extensions/src/app/services/api/video.ts:8-18, 20-32

Backend Processing

The backend is responsible for validating, processing, and storing the uploaded media.

Video Processing and Storage

The VideoService contains the core logic for handling video uploads. The process involves correcting potential issues with browser-recorded videos before storing them.

1. Receive Upload: The controller receives a video file (Express.Multer.File).
2. FFMPEG Correction: The service uses fixVideoWithFFMPEG to process the video. This step is crucial for fixing duration metadata and ensuring compatibility, especially for webm files created by MediaRecorder.
3. Upload to Bucket: The corrected video is uploaded to a Firebase Storage bucket using uploadVideoToBucket. The path is structured by user ID (e.g., users/{uid}/videos/{filename}).
4. Database Entry: Metadata about the video (title, duration, storage reference, creation time) is saved to the Firebase Realtime Database under users/{uid}/videos/{videoId} using the addVideoToDb method.

This flowchart shows the backend video upload process.

Gauzy Upload Service

The GauzyUploadService provides a generic mechanism for uploading different media types (VIDEO, PHOTO, AUDIO) to a "Gauzy" backend. It constructs a FormData payload containing the file and associated metadata (e.g., tenantId, organizationId, recordedAt).

This service appears to be for integration with the Gauzy platform, a separate concern from the primary user-facing media storage.

Key methods include:

• video(payload)
• screenshot(payload)
• camshot(payload)
• soundshot(payload)

Each method builds a form and uses a private post helper to send the data.

Sources: apps/api/src/module/gauzy/services/gauzy-upload.service.ts:27-133

Client-side Handling

The frontend clients (portal and extensions) include logic to prepare media for upload.

User Interface for Uploads

Components like UploadScreen and UploadWorkspaceImageModal provide the user interface for uploading files. They support:

• Drag and Drop: Using react-dropzone or native drop events.
• File Picker: A standard <input type="file">.
• Paste from Clipboard: Handling onPaste events to upload images directly from the clipboard.

Before uploading, the client validates the file type and size. For images, it often converts the file to a base64 string using a utility like fileGetBase64 before sending it to the API, especially for newly captured screenshots that exist in memory.

Sources: apps/extensions/src/content/panel/screens/uploadScreen/UploadScreen.tsx:30-100, apps/portal/components/pagesComponents/_imagesScreen/components/uploadWorkspaceImageModal/UploadWorkspaceImageModal.tsx:32-70

Client-side Video Preparation

The useManageVideoData hook is central to preparing recorded videos for upload. When a screen recording is stopped, the browser provides a series of blob URLs. This hook performs the necessary steps to convert them into a single, valid video file.

1. Transform Blobs: transformBlobURLs is called to fetch the data from each blob URL and create an array of Blob objects.
2. Combine Blobs: The array of blobs is combined into a single Blob with the MIME type video/mp4.
3. Fix Duration: The fix-webm-duration library is used to correct the video's duration metadata. This is a critical step because MediaRecorder often writes an incorrect duration, leading to playback issues.
4. Upload: The final, corrected blob is passed to an upload function (uploadVideo or uploadVideoToService) which sends it to the backend API.

This sequence diagram illustrates the flow within the useManageVideoData hook.