Data Upload and Validation
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Data Upload and Validation system is a cornerstone of the BMSview application, responsible for ingesting user-provided Battery Management System (BMS) screenshots and ensuring the integrity of the data extracted by the AI. This process involves a client-side upload interface, a serverless backend for analysis, a sophisticated data validation layer, and an autonomous self-correction mechanism.
The primary goal is to provide a seamless user experience for submitting data while rigorously validating the extracted information against physical laws and logical constraints. This ensures that all downstream features, such as the Ultimate AI Battery Guru, operate on accurate and reliable data. The system supports both single-file synchronous uploads for immediate feedback and bulk uploads.
System Architecture and Flow
The process begins when a user uploads one or more BMS screenshots through the web interface. The frontend sends the image data to a serverless function, which orchestrates the AI-based data extraction, validation, and storage.
The following diagram illustrates the high-level data flow from upload to final storage.
Server-Side Analysis and Validation
The core logic resides in the backend, primarily within the analyze.cjs Netlify function and its associated utilities. When an image is received, it enters an analysis pipeline that includes several key stages.
Autonomous Self-Correction
A key feature of the validation system is its ability to autonomously detect and correct errors from the AI's data extraction process. This is achieved through a validation-driven retry loop.
- Initial Extraction: The AI attempts to extract data from the image.
- Validation & Scoring: The extracted data is passed to the
validateAnalysisData()function. This function checks the data against numerous rules and calculates a quality score from 0 to 100. - Decision:
- If the score is high (e.g., ≥ 80) or acceptable (e.g., ≥ 60), the data is considered valid and the process continues.
- If the score is low, the system generates specific, actionable feedback for the AI, explaining the validation errors.
- Retry: The system re-submits the image and the feedback to the AI for another attempt. This loop can run up to 3 times.
- Finalization: If the loop completes without achieving a satisfactory score, the system saves the best attempt and flags the record for manual review.
Sources: AUTONOMOUS_SELF_CORRECTION.md, SANITY_CHECK_REPORT.md
The sequence diagram below shows the interaction during a self-correction cycle.
Sources: AUTONOMOUS_SELF_CORRECTION.md
Data Validation Rules
The validateAnalysisData() function, defined in netlify/functions/utils/data-validation.cjs, enforces a comprehensive set of rules. These are categorized as physics checks, range checks, and logical consistency checks.
| Validation Check | Description | Criticality | Source File |
|---|---|---|---|
| State of Charge (SOC) | Ensures SOC is within the valid range of 0-100%. | Critical | tests/data-validation.test.js |
| Cell Voltage Range | Validates each cell voltage is between 2.0V and 4.5V. | Critical | tests/data-validation.test.js |
| Physics: Voltage Sum | Verifies that the sum of individual cell voltages equals the overall voltage (±0.5V tolerance). | Critical if > 1.0V diff | SANITY_CHECK_REPORT.md |
| Temperature Range | Checks that battery temperatures are between 0°C and 100°C. Flags 0°C as suspicious. | Critical | SANITY_CHECK_REPORT.md |
| Physics: Power Calculation | Confirms that Power ≈ Current × Voltage (±10% tolerance). | Critical if > 50% diff | SANITY_CHECK_REPORT.md |
| Capacity Consistency | Ensures remaining capacity does not exceed full capacity (with a 5% tolerance). | Critical | SANITY_CHECK_REPORT.md |
| SOC vs. Capacity | Checks if reported SOC matches the calculated ratio of remaining/full capacity (within 10%). | Critical if ≥ 25% diff | SANITY_CHECK_REPORT.md |
| MOS Logic | Checks if discharge/charge MOS states are consistent with the current's direction. | Warning | SANITY_CHECK_REPORT.md |
| Cell Voltage Statistics | Verifies that reported highest, lowest, and average cell voltages match calculations from the cell array. | Warning | SANITY_CHECK_REPORT.md |
Sources: tests/data-validation.test.js, SANITY_CHECK_REPORT.md, DATA_VALIDATION_IMPLEMENTATION.md
Smart Duplicate Handling
The system employs a quality-based approach to handling duplicate image uploads. When an image with an existing SHA-256 hash is uploaded, the system checks the quality score of the previously stored analysis.
This flow ensures that if a better analysis can be performed (e.g., due to AI model improvements), the existing record is "upgraded" to the higher-quality version.
Sources: AUTONOMOUS_SELF_CORRECTION.md
Database Schema for Validation
To support the validation and review process, several fields have been added to the analysis-results collection in MongoDB.
| Field Name | Type | Description |
|---|---|---|
needsReview | Boolean | true if the final validation score was low, flagging it for manual review. |
validationWarnings | Array of Strings | A list of all warnings generated during the validation process. |
validationScore | Number | The final quality score (0-100) of the extracted data. |
extractionAttempts | Number | The number of retries the self-correction loop performed. |
_wasUpgraded | Boolean | true if this record was upgraded from a lower-quality duplicate. |
_previousQuality | Number | The quality score before an upgrade. |
_newQuality | Number | The quality score after an upgrade. |
Sources: AUTONOMOUS_SELF_CORRECTION.md, DATA_VALIDATION_IMPLEMENTATION.md
Conclusion
The Data Upload and Validation system is a critical component that safeguards the quality of data within BMSview. By combining a multi-stage validation process with an innovative autonomous self-correction loop, the system maximizes the accuracy of AI-extracted data. This robust foundation ensures that users receive reliable, physically-possible, and consistent analytics, which is essential for the advanced insights and recommendations provided by the application. The quality-aware duplicate handling further enhances the data corpus over time by replacing older, lower-quality analyses with improved versions.