AR Virtual Try-On

Relevant source files

The following files were used as context for generating this wiki page:

• features/ar/hooks/use-face-detector.ts
• features/ar/hooks/use-glasses-overlay.ts
• features/ar/components/virtual-tryon-dialog.tsx
• app/(account)/try-on/test/page.tsx/try-on/test/page.tsx)
• features/ar/types/glasses-try-on.types.ts
• types/product.types.ts
• features/products/schemas/product-form.schema.ts
• app/admin/products/new/page.tsx
• app/admin/products/[id]/edit/page.tsx
• features/ar/components/product-selector.tsx

The AR Virtual Try-On feature provides users with an interactive way to see how glasses look on them using their device's webcam. This is a purely client-side feature that leverages the face-api.js library for real-time face and landmark detection. It works by capturing a user's photo, identifying key facial features (specifically the eyes), and then overlaying a transparent PNG image of the selected glasses product onto the photo.

The feature is enabled on a per-product basis through the admin panel, where an administrator can upload a specific "try-on image" for each product. This modular system is composed of React hooks for logic, UI components for user interaction, and specific data types to ensure consistency.

System Architecture

The Virtual Try-On system is composed of several key client-side components and hooks that work together. The main user interface is the VirtualTryonDialog, which orchestrates the process. It uses the WebcamCapture component to get an image from the user, the useFaceDetector hook to find facial landmarks, and the useGlassesOverlay hook to render the final image with glasses. The ProductSelector component allows the user to switch between different glasses.

The diagram below illustrates the high-level architecture of the feature.

Core Components & Hooks

The feature is primarily built around two custom hooks for processing and a set of components for the user interface.

Name	Type	Description	Source File
useFaceDetector	Hook	Manages loading face-api.js models and detecting facial landmarks from a captured image.	features/ar/hooks/use-face-detector.ts
useGlassesOverlay	Hook	Takes a user's image, face landmarks, and a glasses image URL to generate a new image with the glasses overlaid in the correct position.	features/ar/hooks/use-glasses-overlay.ts
WebcamCapture	Component	Provides the webcam feed and a button to capture a still image. Returns the image as a base64 data string.	app/(account)/try-on/test/page.tsx:237-243
ProductSelector	Component	Displays a carousel of products that support virtual try-on, allowing the user to select one.	features/ar/components/product-selector.tsx
VirtualTryonDialog	Component	The main dialog that contains all other components and orchestrates the entire try-on flow.	features/ar/components/virtual-tryon-dialog.tsx

Data Flow and User Interaction

The user interaction follows a clear, step-by-step process from capturing an image to seeing the final result. The test page app/(account)/try-on/test/page.tsx provides a complete implementation of this flow.

1. The user opens the Virtual Try-On dialog.
2. The WebcamCapture component is initialized, requesting camera access.
3. The user clicks the "Capture" button.
4. The onCapture callback is triggered with a base64 image string.
5. The handleCapture function calls detectFace from the useFaceDetector hook.
6. If a face is found, the FaceLandmarks are stored in the component's state.
7. The user selects a product from the ProductSelector.
8. The tryOnImageUrl of the selected product is retrieved.
9. The overlayGlasses function from the useGlassesOverlay hook is called with the captured image, the glasses image URL, and the face landmarks.
10. The hook returns a new base64 image string with the glasses overlaid, which is then displayed to the user.

The following sequence diagram details this interaction:

Face Detection (useFaceDetector)

This hook is responsible for all face detection logic. It uses face-api.js to perform its tasks.

• Model Loading: On initialization, it loads two required models from the /public/models/ directory:
  • tinyFaceDetector: For fast face detection.
  • faceLandmark68Net: For detecting 68 specific points on the face.
• Detection: The detectFace function takes an HTMLImageElement and uses the loaded models to find a face. If successful, it extracts the coordinates of the left and right eyes and the detection confidence score.
• Output: It returns a FaceLandmarks object or null if no face is detected.

The face detection process is as follows:

Glasses Overlay (useGlassesOverlay)

This hook handles the image composition. It takes the original image, the glasses image, and the detected landmarks to create the final result.

• Calculation: It calculates the center point between the eyes, the distance between them (to determine scale), and the angle of the eyes (to determine rotation).
• Canvas Drawing: It creates an in-memory <canvas> element.
  1. It first draws the user's captured image onto the canvas.
  2. It then calculates the correct position, width, height, and rotation for the glasses image based on the eye landmarks.
  3. Using ctx.translate, ctx.rotate, and ctx.drawImage, it draws the glasses image onto the canvas at the calculated position.
• Output: It returns the canvas content as a base64 PNG data URL.

Sources: features/ar/hooks/use-glasses-overlay.ts

Data Models and Types

The feature relies on a set of strictly-typed interfaces to ensure data consistency between the different parts of the system.

Interface	Description	Source File
ProductWithTryOn	Represents a product that supports the try-on feature. It must have a tryOnImageUrl.	features/ar/types/glasses-try-on.types.ts:10-16
FaceLandmarks	Contains the x/y coordinates for the left and right eyes, an optional nose coordinate, and the confidence score of the detection.	features/ar/types/glasses-try-on.types.ts:24-30
GlassesOverlayConfig	Defines the calculated properties needed to draw the glasses on the canvas, including position, scale, rotation, width, and height.	features/ar/types/glasses-try-on.types.ts:44-51
TryOnError	A custom error class used to signal specific failures within the try-on flow, such as NO_FACE_DETECTED or MODEL_LOAD_FAILED.	features/ar/types/glasses-try-on.types.ts:98-104

Sources: features/ar/types/glasses-try-on.types.ts

Product Integration

For a product to be available in the Virtual Try-On feature, it must have a dedicated try-on image uploaded via the admin panel.

• Admin Forms: Both the "Create Product" and "Edit Product" pages in the admin section (/admin/products/new and /admin/products/[id]/edit) include a file input field specifically for the "Try-on Image".
• Data Types: The AdminCreateProductRequest and AdminUpdateProductRequest types include an optional tryOnImage: File property.
• Validation: The productFormSchema uses a specific tryOnImageFileSchema to validate the uploaded file. This ensures the file is a PNG and does not exceed the 20MB size limit.

Sources: app/admin/products/new/page.tsx:32, app/admin/products/[id]/edit/page.tsx:43, types/product.types.ts:153, features/products/schemas/product-form.schema.ts:21-31

This integration is critical, as the tryOnImageUrl is what links a product in the database to the AR functionality on the frontend. The useProductsWithTryOn hook fetches only those products that have this URL available.

Sources: app/(account)/try-on/test/page.tsx:46