LLM Vision Integration

Relevant source files

The following files were used as context for generating this wiki page:

• server/_core/llm.ts
• README.md
• todo.md
• drizzle/schema.ts
• client/src/components/AIChatBox.tsx
• server/db.ts

Introduction

The LLM Vision Integration is a core architectural component of Driveway Estimator Pro, enabling automated processing of visual data to simplify the estimation workflow. Its primary purpose is to handle multimodal requests—combining text instructions with image data—to perform complex spatial analysis and photorealistic rendering.

This system is utilized for two main high-level features: AI Edge Detection, which identifies driveway boundaries to calculate square footage, and Material Visualization, which uses image generation to provide users with a preview of how different asphalt or gravel materials will look on their specific property.
Sources: README.md:7-12, todo.md:14-17

Architecture and Core Implementation

The integration is built around a centralized invocation service that proxies requests to the Forge API. The system is designed to be type-safe and handles the normalization of messages into a format compatible with advanced multimodal models like gemini-2.5-flash.

Multimodal Message Handling

The core logic resides in server/_core/llm.ts, which defines a strict structure for messages containing text, images, or files. This allows the backend to send raw driveway photos along with prompts asking for boundary coordinate detection.

The invokeLLM Service

The invokeLLM function is the primary entry point for all AI vision tasks. It performs the following steps:

1. API Validation: Ensures the Forge API key is configured.
2. Message Normalization: Converts multimodal content parts (text and images) into standardized payloads.
3. Schema Enforcement: Configures json_schema response formats to ensure the LLM returns structured data (like coordinate arrays) rather than raw text.
4. Hardware Optimization: Configures a thinking budget and specific model parameters for gemini-2.5-flash.

Sources: server/_core/llm.ts:183-228

Vision Processing Flows

The system supports two distinct data flows for vision-related tasks: analysis (Edge Detection) and generation (Material Preview).

AI Edge Detection Flow

This flow converts a user-captured photo into a set of geometric coordinates used for area calculation.

Material Visualization Flow

This system uses generative AI to create a photorealistic rendering of the driveway with the selected material overlay.
Sources: todo.md:27, README.md:10-12

Data Structures and Schema Integration

The output from LLM vision processes is persisted in the projects table. Structured AI output is mapped directly to specific database fields to allow for subsequent manual adjustments by the user.

Project Schema Fields (AI Related)

Field	Type	Description
photoUrl	text	The original source image sent to the LLM for analysis.
cornerPoints	text (JSON)	Coordinates detected by the LLM Vision model.
squareFeet	int	Area derived from the AI-detected cornerPoints.
previewImageUrl	text	URL of the AI-generated photorealistic material render.

Sources: drizzle/schema.ts:45-66

Structured Output Configuration

To ensure reliability, the integration uses ResponseFormat objects to force the LLM into returning specific JSON structures.

// Example of the response format used for vision extraction
export type ResponseFormat =
  | { type: "text" }
  | { type: "json_object" }
  | { type: "json_schema"; json_schema: JsonSchema };

Sources: server/_core/llm.ts:108-111

UI Integration: AIChatBox

While edge detection is an automated background process, the project also includes an AIChatBox component that demonstrates how users can interact with the LLM. This component supports streaming responses via the Streamdown library and handles the display of AI-generated content.
Sources: client/src/components/AIChatBox.tsx:142-167

Conclusion

The LLM Vision Integration transforms static image data into actionable project metrics. By bridging the gap between raw pixel data and the Drizzle ORM schema via the Forge API, the system enables features like automated measurement and visual previews that traditionally require manual input or specialized hardware like LiDAR.
Sources: README.md:7-12, todo.md:14-27