Feature: AI Edge Detection
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The AI Edge Detection feature is a core component of the Driveway Estimator Pro system, designed to automate the measurement of driveway surface areas. It utilizes LLM Vision analysis to process captured photos and identify the boundary coordinates (corners) of a driveway. This technology replaces manual measurement tools with an intelligent computer vision approach, allowing homeowners and contractors to obtain square footage estimates in seconds.
The system workflow begins with a camera capture or photo upload, followed by server-side vision analysis. Once the AI identifies the edges, the system calculates the square footage based on image dimensions and detected corner points. Users retain the ability to fine-tune these measurements through a manual adjustment UI before proceeding to material selection and pricing.
Sources: README.md:12-14, README.md:46, todo.md:14
Architecture and Data Flow
The feature follows a client-server architecture where the frontend handles image acquisition and visualization, while the backend performs the heavy lifting of vision analysis and geometric calculations.
System Sequence
The following diagram illustrates the lifecycle of an edge detection request from the initial user upload to the final square footage calculation.
The sequence begins in the frontend Estimator.tsx and communicates via the projects.uploadPhotoAndDetectEdges procedure.
Sources: server/routers/projects.ts:70-108, client/src/pages/Estimator.tsx:112-143
Technical Components
API Endpoint: uploadPhotoAndDetectEdges
This protected tRPC procedure is the primary entry point for the edge detection feature. It manages the orchestration between storage, AI vision, and area calculation.
| Parameter | Type | Description |
|---|---|---|
photoBase64 | string | The image data encoded in base64 format. |
photoName | string | Original name of the uploaded file. |
imageWidth | number | The pixel width of the original image. |
imageHeight | number | The pixel height of the original image. |
Returns:
photoUrl: The public S3 URL of the uploaded photo.corners: An array of{x, y}coordinates representing driveway boundaries.squareFeet: The calculated area in square feet.confidence: AI confidence score for the detection.
Sources: server/routers/projects.ts:70-82, server/routers/projects.ts:101-106
Area Calculation Logic
The backend uses the detected corner points to compute the surface area. The function calculateSquareFeetFromCorners takes normalized corner coordinates (percentage-based) and translates them into physical dimensions using image metadata and calibration factors.
Key logic handled by the service includes:
- Shape Handling: Processing rectangular, trapezoidal, and non-rectangular driveway shapes.
- Calibration: Utilizing a
pixelsPerFootfactor to convert image units to imperial measurements. - Constraints: Implementing a minimum threshold (e.g., 100 sq ft) to prevent anomalous results from small detections.
Sources: server/services/edgeDetection.test.ts:7-25, server/services/edgeDetection.test.ts:51-70
Frontend Manual Adjustment
The Estimator component provides a "Step 2: Adjust Driveway Corners" interface. This allows users to correct any inaccuracies in the AI's detection by dragging interactive markers over the image.
The UI uses relative positioning for the image container and absolute positioning for markers based on percentage coordinates (e.g., left: {corner.x}%).
Sources: client/src/pages/Estimator.tsx:162-181, client/src/pages/Estimator.tsx:327-348
Data Structures
CornerPoint
The fundamental data structure used for representing boundaries throughout the system.
interface CornerPoint {
x: number; // 0-100 (percentage of width)
y: number; // 0-100 (percentage of height)
}
Sources: client/src/pages/Estimator.tsx:17-20, server/routers/projects.ts:175
Detection Result
The object returned by the LLM Vision analysis service.
interface EdgeDetectionResult {
corners: CornerPoint[];
confidence: number;
description: string;
}
Sources: server/routers/projects.ts:98-100
Implementation Details
Image Processing
Before analysis, images are converted from base64 strings to Buffers and stored in S3 with unique keys structured as projects/{userId}/{nanoid}-{filename}. This URL is then passed to the Vision LLM to avoid sending large base64 payloads repeatedly.
Sources: server/routers/projects.ts:86-94
Geometric Validation
Tests ensure the edge detection service correctly handles various real-world scenarios:
- Rectangle Calculation: Validating standard 4-point areas.
- Scale Sensitivity: Ensuring that higher calibration factors or larger image dimensions result in proportionally larger area calculations.
- Non-Rectangular Shapes: Using trapezoidal coordinate sets to verify the robustness of the area algorithm.
Sources: server/services/edgeDetection.test.ts:72-108
Summary
The AI Edge Detection system integrates LLM Vision with geometric processing to provide automated driveway measurements. By combining server-side AI analysis with a client-side adjustment interface, the feature ensures both ease of use and high accuracy for project estimations. This module is the foundation for the project's downstream pricing and material visualization features.