Storage & Data Processing

Relevant source files

The following files were used as context for generating this wiki page:

• src/common/utils/storage.ts
• src/common/utils/dataProcessing.ts
• tests/unit/utils/dataProcessing.test.ts
• tests/unit/utils/storage.test.ts
• src/core/ai/PricePredictionEngine.ts
• src/common/models/unifiedClient/UnifiedBrowserManager.ts

The Storage & Data Processing system serves as the foundational persistence and transformation layer for MarketSage. It is responsible for managing local data synchronization via Chrome's storage API, maintaining product caches, tracking price histories, and normalizing raw marketplace data for downstream consumption by AI models and user interfaces.

This system ensures that high-frequency data, such as real-time price updates and seller ratings, is efficiently stored with automatic cache eviction policies to maintain optimal browser performance. It also provides the necessary utility functions to clean and standardize inconsistent data formats encountered across various Chinese marketplaces like Temu and AliExpress.

Storage Architecture

The storage layer is implemented through a centralized Storage class that abstracts interactions with chrome.storage.local. It utilizes a key-value structure where specific prefixes categorize different types of application data.

Key Storage Components

Component	Description	Persistence Strategy
Price History	Tracks historical price points for specific products with ISO timestamps.	Local storage with product ID suffix.
Product Cache	Stores the last 100 viewed products to enable rapid retrieval.	LRU-style eviction (keeps 100 most recent).
Seller Ratings	Aggregates and calculates average ratings for marketplace sellers.	Cumulative total and count tracking.
Unified Cart	Manages a cross-platform shopping cart for unified checkout.	Single array of CartItem objects.
Model Data	Persists TensorFlow.js model topologies and weights.	Binary/JSON blobs in local storage.

Sources: [src/common/utils/storage.ts:4-13](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/common/utils/storage.ts#L4-L13), [src/common/utils/storage.ts:41-53](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/common/utils/storage.ts#L41-L53), [src/core/ai/PricePredictionEngine.ts:316-324](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/core/ai/PricePredictionEngine.ts#L316-L324)

Data Flow Diagram

The following diagram illustrates how data flows from marketplace extraction through storage and into the prediction engines.

Data Processing & Normalization

The DataProcessing utility (and its associated tests) provides logic for cleaning marketplace data. This is critical for handling varied currency formats, white-space heavy titles, and inconsistent image URL arrays.

Key Normalization Functions

• normalizePrice: Formats numeric values into standard strings with currency symbols and two decimal places (e.g., USD 10.50).
• extractNumericPrice: Parses strings like $1,234.56 or €1.234,56 into pure floating-point numbers, handling locale-specific thousand separators.
• calculateDiscount: Determines percentage drops between original and current prices, rounded to the nearest integer.
• normalizeSpecifications: Trims keys and values in product attribute maps to ensure consistent object keys for comparison.

Sources: [tests/unit/utils/dataProcessing.test.ts:4-30](https://github.com/VpkDevs/MarketSage/blob/HEAD/tests/unit/utils/dataProcessing.test.ts#L4-L30), [tests/unit/utils/dataProcessing.test.ts:70-79](https://github.com/VpkDevs/MarketSage/blob/HEAD/tests/unit/utils/dataProcessing.test.ts#L70-L79)

Price Parsing Logic

Cache Management

To prevent storage bloat, the Product Cache implements a limit of 100 entries. When a new product is cached using Storage.cacheProduct(), the system sorts existing entries by their timestamp and retains only the most recent 100.

Cache Structure

interface CachedProduct extends Product {
  timestamp: string; // ISO String
}

// Storage Key: 'product_cache'
type ProductCache = Record<string, CachedProduct>;

Sources: [src/common/utils/storage.ts:41-45](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/common/utils/storage.ts#L41-L45), [tests/unit/utils/storage.test.ts:48-58](https://github.com/VpkDevs/MarketSage/blob/HEAD/tests/unit/utils/storage.test.ts#L48-L58)

Transactional State Management

The UnifiedBrowserManager utilizes storage to maintain the state of cross-platform checkout sessions. This ensures that even if the browser is closed, the progress of a multi-platform purchase remains intact.

Checkout Session Schema

Field	Type	Description
sessionId	string	Unique identifier for the transaction.
platforms	PlatformCheckout[]	Status of individual platform checkouts (e.g., AliExpress, Temu).
status	string	Overall session status: started, in-progress, completed, failed.
startedAt	number	Unix timestamp of session initiation.

Sources: [src/common/models/unifiedClient/UnifiedBrowserManager.ts:45-56](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/common/models/unifiedClient/UnifiedBrowserManager.ts#L45-L56), [src/common/utils/storage.ts:108-115](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/common/utils/storage.ts#L108-L115)

Unified Cart Sequence

AI Model Storage

The PricePredictionEngine interacts with storage to save and load neural network artifacts. This prevents the need to retrain models on every extension load.

• saveModelData: Stores the model topology, weight specifications, and actual weight data.
• getModelData: Retrieves the stored artifacts. The engine checks the version field (e.g., 1.0) to ensure compatibility before loading the model into TensorFlow.js.

Sources: [src/core/ai/PricePredictionEngine.ts:54-65](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/core/ai/PricePredictionEngine.ts#L54-L65), [src/core/ai/PricePredictionEngine.ts:316-324](https://github.com/VpkDevs/MarketSage/blob/HEAD/src/core/ai/PricePredictionEngine.ts#L316-L324)

The Storage and Data Processing module provides a robust foundation for handling marketplace data. By combining strict normalization routines with an asynchronous, limited-capacity storage layer, MarketSage maintains data integrity across various shopping platforms while ensuring efficient resource usage within the browser environment.