Feature: Image Processing and Analysis

Relevant source files

The following files were used as context for generating this wiki page:

• src/lib/image-processing.ts
• context/PRD.md
• package.json
• context/TYPES_AND_INTERFACES.md
• AGENTS.md
• CLAUDE.md

The Image Processing and Analysis feature is a core component of VineSight's AI-powered smart farming capabilities. It provides the foundational tools for computer vision tasks, enabling real-time plant health assessment and disease identification directly on the user's device. This system is designed to take raw images, typically from a mobile camera, and transform them into a format suitable for machine learning models, as well as extract valuable analytical data.

This feature leverages client-side technologies to ensure fast, offline-capable analysis. The primary goal is to support features like AI-Powered Disease Detection and Plant Health Analysis by providing a standardized pipeline for image loading, pre-processing, enhancement, and data extraction. The architecture is centered around the ImageProcessingService, which encapsulates all image manipulation logic.

Sources: context/PRD.md, AGENTS.md, src/lib/image-processing.ts

Architecture and Core Technologies

The image processing system is built to run entirely on the client-side, utilizing a combination of browser APIs and specialized JavaScript libraries. This approach minimizes server load and allows for real-time feedback and offline functionality.

The main technologies used are:

• TensorFlow.js: The core machine learning framework for running computer vision models (e.g., CNNs for disease classification) in the browser.
• Canvas API: Used for low-level image manipulation, including resizing, filtering, and pixel data extraction.
• React Webcam: A component likely used for capturing images directly from the device's camera.
• Web Workers (Implied): For offloading heavy computations to background threads to keep the UI responsive, although not explicitly detailed in the provided files.

Sources: package.json:28, AGENTS.md, context/PRD.md

The following diagram illustrates the technology stack for the image analysis feature.

Image Processing Pipeline

The ImageProcessingService provides a standardized pipeline to prepare images for AI analysis. The processImageForAI method is the main entry point for this workflow.

The pipeline consists of the following steps:

1. Load Image: An image is loaded from a File, Blob, or URL string into an HTMLImageElement.
2. Extract Metadata: Basic metadata like dimensions, file size, and type is extracted.
3. Calculate Dimensions: The image is resized to fit within specified maximum dimensions (defaulting to 224x224 pixels) while maintaining its aspect ratio. This is a common input size for computer vision models.
4. Draw to Canvas: The resized image is drawn onto an HTML canvas, which serves as the workspace for all subsequent manipulations.
5. Pre-processing (Optional):
   • Enhancement: If the image is too dark or bright, contrast and brightness are automatically adjusted.
   • Normalization: Pixel values are prepared for the AI model.
6. Output: The service returns a ProcessedImage object containing the original image, the processed canvas, metadata, and a summary of applied pre-processing steps.

Sources: src/lib/image-processing.ts:32-84

This diagram shows the data flow within the processImageForAI function.

Key Components

ImageProcessingService

This is a static class that acts as a utility for all image-related operations. It does not need to be instantiated. It is initialized on module load to create a shared canvas element for processing.

Sources: src/lib/image-processing.ts:25-29

Data Structures

Several interfaces define the data structures used throughout the image processing feature.

Interface	Description	Fields
ImageProcessingOptions	Configuration for the processImageForAI function.	maxWidth, maxHeight, quality, format, enablePreprocessing
ImageMetadata	Stores metadata extracted from the original image file.	width, height, size, format, timestamp, location?
ProcessedImage	The output object from the main processing pipeline.	originalImage, processedImage, metadata, preprocessing
DiseaseDetectionResult	Defines the structure for the output of the disease detection AI model.	disease, confidence, severity, treatmentRecommendations, preventionMeasures

Sources: src/lib/image-processing.ts:1-23, context/TYPES_AND_INTERFACES.md

Core Functionalities

AI Pre-processing

The processImageForAI function is central to preparing images for machine learning models.

// src/lib/image-processing.ts:32-36
static async processImageForAI(
    imageFile: File | Blob | string,
    options: ImageProcessingOptions = {},
): Promise<ProcessedImage> {

• Resizing: Images are resized to a standard dimension (e.g., 224x224) to match the expected input of neural networks. The calculateDimensions method preserves the aspect ratio.
• Enhancement: The enhanceImage method adjusts brightness and contrast if needsEnhancement detects the image is too dark (average brightness < 100) or too bright (average brightness > 200).
• Normalization: The normalizeForAI method scales pixel values, which is a standard requirement for neural network inputs.

Sources: src/lib/image-processing.ts:32-84, 131-150, 172-193

Plant Health Analysis

The extractColorStatistics function analyzes the pixel data of a processed image to calculate health indices. This provides a quantitative measure of plant health without requiring a full AI model.

// src/lib/image-processing.ts:251-257
static extractColorStatistics(image: HTMLCanvasElement): {
    averageRGB: { r: number; g: number; b: number }
    greenHealthIndex: number // 0-100
    browningIndex: number // 0-100
    yellowingIndex: number // 0-100
} {

The following health indices are calculated:

Index	Formula	Description
greenHealthIndex	(green_pixels / total_pixels) * 100	Represents the percentage of healthy green tissue in the image.
browningIndex	(brown_pixels / total_pixels) * 100	Represents the percentage of brown (potentially diseased or dead) tissue.
yellowingIndex	(yellow_pixels / total_pixels) * 100	Represents the percentage of yellowing tissue, an indicator of stress.

Sources: src/lib/image-processing.ts:251-300, context/PRD.md

Disease Detection Support

Several functions are specifically designed to improve the accuracy of disease detection models:

• applyPlantAnalysisFilters: This function enhances the green channel and red/brown tones to increase the contrast between healthy and diseased tissue, making it easier for a model to identify symptoms.
• cropToPlantArea: This automatically crops the image to the detected plant material, removing irrelevant background and focusing the AI model's attention on the area of interest. It identifies green pixels to determine the bounding box of the plant.

Sources: src/lib/image-processing.ts:223-250, 329-388

TensorFlow.js Integration

The final step before feeding an image to a model is converting it to a tensor. The canvasToTensor method handles this by iterating through the canvas's pixel data and creating a 3D array ([height][width][channels]) with RGB values normalized between 0 and 1.

// src/lib/image-processing.ts:303-326
static canvasToTensor(canvas: HTMLCanvasElement): number[][][] {
    const context = canvas.getContext('2d')!
    const imageData = context.getImageData(0, 0, canvas.width, canvas.height)
    const data = imageData.data

    const tensor: number[][][] = []

    for (let y = 0; y < canvas.height; y++) {
      const row: number[][] = []
      for (let x = 0; x < canvas.width; x++) {
        const index = (y * canvas.width + x) * 4
        const pixel = [
          data[index] / 255, // R
          data[index + 1] / 255, // G
          data[index + 2] / 255, // B
        ]
        row.push(pixel)
      }
      tensor.push(row)
    }

    return tensor
}

Summary

The Image Processing and Analysis feature is a critical enabler for VineSight's smart farming capabilities. By providing a robust, client-side pipeline for preparing and analyzing images, it allows for the integration of advanced computer vision models directly within the user's browser. This system supports key functionalities such as disease detection, plant health monitoring, and other visual assessments, delivering immediate, actionable insights to farmers without reliance on server-side processing.

Sources: src/lib/image-processing.ts, context/PRD.md, AGENTS.md