Data Flow

The data flow in BMSview is designed around a local-first architecture, where the browser's IndexedDB is the primary data store for the user to ensure offline functionality and a responsive UI. The server, powered by Netlify Functions, acts as the authoritative source and handles complex processing tasks like AI analysis, data aggregation, and integrity checks. Data is synchronized periodically between the client and server.

This document outlines the primary data pathways within the application, including BMS screenshot analysis, historical data retrieval for charting, AI-driven insights generation, and administrative data reconciliation. The system leverages a combination of synchronous API calls for immediate feedback and asynchronous background jobs for more intensive operations.

Sources: GEMINI.md

BMS Screenshot Analysis Flow

This flow describes the process from a user uploading a BMS screenshot to receiving an analysis. The application primarily uses a synchronous flow for single-file uploads to provide a fast user experience.

Sources: GEMINI.md

The diagram below illustrates the sequence of events for a synchronous analysis request.

This sequence shows the synchronous analysis flow for a single screenshot upload.
Sources: GEMINI.md, STATE_MANAGEMENT_GUIDE.md, SANITY_CHECK_REPORT.md

Key Steps:

1. Upload: The user uploads a screenshot. The clientService dispatches a PREPARE_ANALYSIS action to update the UI state.
2. API Call: A synchronous request is made to the /.netlify/functions/analyze endpoint.
3. Deduplication: The backend calculates a SHA-256 hash of the image content to check if an analysis already exists in the analysis-results collection.
4. AI Analysis: If no record is found, the image is sent to the Google Gemini API for OCR and analysis.
5. Data Validation: The extracted data is passed through a data integrity validation layer (validateAnalysisData) which performs physics and consistency checks (e.g., SOC range, voltage sum).
6. Storage: The new analysis result, along with any validation warnings and a needsReview flag, is stored in the analysis-results collection in MongoDB.
7. Response: The complete AnalysisRecord is returned to the client.
8. UI Update: The clientService dispatches a SYNC_ANALYSIS_COMPLETE action, which updates the appState and displays the results to the user.

Sources: GEMINI.md, SANITY_CHECK_REPORT.md, 12-2-todo.md

Historical & Analytics Data Flow

The Historical Analysis feature provides administrators with advanced charting and analytics. This system can operate in two modes: displaying only BMS screenshot data or displaying a merged timeline of BMS data and hourly cloud/weather data.

Sources: ADMIN_HISTORICAL_ANALYSIS_OVERHAUL.md, HOURLY_CLOUD_SOLAR_INTEGRATION.md

The following flowchart shows the data retrieval logic for the historical chart.

This diagram illustrates the two data paths for populating the historical analysis chart.
Sources: HISTORICAL_ANALYSIS_IMPLEMENTATION_COMPLETE.md, ADMIN_HISTORICAL_ANALYSIS_OVERHAUL.md

Data Retrieval Modes

1. BMS-Only Mode

• The HistoricalChart.tsx component, via the useAnalyticsData.ts hook, fetches data from the standard /.netlify/functions/history endpoint.
• This endpoint returns an array of AnalysisRecord objects directly from the history collection in MongoDB for the selected system and date range.

Sources: HISTORICAL_ANALYSIS_IMPLEMENTATION_COMPLETE.md

2. Merged Data Mode

• When the "Hourly Weather" toggle is enabled, the frontend calls the getMergedTimelineData function.
• This function hits the /.netlify/functions/history endpoint with the merged=true query parameter.
• The backend then executes the mergeBmsAndCloudData utility from data-merge.cjs. This utility:
  1. Fetches BMS records from the history collection.
  2. Fetches hourly weather data (cloud cover, irradiance) from the hourly-weather collection.
  3. Performs linear interpolation to estimate BMS metric values at hourly intervals where no screenshot exists.
  4. Merges the two datasets, prioritizing real BMS data where available.
  5. Tags each data point with its source: 'bms', 'cloud', or 'estimated'.
  6. Downsamples the final dataset to a maximum of 2000 points for performance if necessary.
• The HistoricalChart.tsx component uses the source field to apply different visual styles (e.g., solid lines for 'bms', dashed for 'estimated').

Sources: ADMIN_HISTORICAL_ANALYSIS_OVERHAUL.md, HOURLY_CLOUD_SOLAR_INTEGRATION.md

AI Insights Generation (ReAct Loop)

For generating deep insights, the system uses a ReAct (Reasoning and Acting) loop with the Gemini AI. This allows the AI to use a set of predefined "tools" to gather more data on-demand to answer complex user queries.

The diagram below shows the ReAct loop, including the mechanism for handling timeouts and resuming jobs.

This sequence illustrates the AI insights generation process, including tool use and the timeout/resume mechanism.
Sources: SESSION_COMPLETE.md, COMPLETION_SUMMARY.md, FINAL_REPORT_TASK_COMPLETION.md

Key Aspects:

• Full Historical Context: Before starting the loop, insights-guru.cjs queries the database for the actual available data range and can load a 90-day daily rollup with hourly averages to give Gemini comprehensive context.
• Tool Calling: Gemini is instructed to use tools like request_bms_data to query the database for specific date ranges if the initial context is insufficient. It is explicitly forbidden from merely recommending tool use to the user.
• Timeout and Resume: For long-running analyses, the synchronous request may time out. The API returns a 408 Timeout status with a jobId. The clientService automatically retries the request, passing the jobId to resume the conversation from its last checkpoint.
• Format Validation: The final response from Gemini is automatically validated. If a user requested a specific format (like CSV or a Markdown table) and the response is malformed, the system sends a correction prompt back to Gemini to fix the formatting before returning it to the user.

Sources: IMPLEMENTATION_COMPLETE.md, INSIGHTS_TOOL_EXECUTION_ENHANCEMENT.md, FINAL_REPORT_TASK_COMPLETION.md

Admin Data Reconciliation

This flow allows administrators to audit data integrity, specifically to find analysis records associated with a Data Logger ID (DL-#) that are not linked to a registered BMS system.

This diagram shows the process for auditing and reconciling orphaned data logger records.
Sources: DATA_RECONCILIATION_PR_SUMMARY.md

Reconciliation Steps:

1. The ReconciliationDashboard.tsx component calls the /.netlify/functions/admin-data-integrity endpoint.
2. This function executes a MongoDB aggregation pipeline on the analysis-results collection.
3. The pipeline groups all records by dlNumber and checks if a corresponding system exists in the systems collection.
4. It returns a list of all dlNumber sources, categorized as MATCHED (linked to a system) or ORPHAN (not linked).
5. The frontend displays orphaned sources in a dedicated table, from which an admin can "adopt" the data by creating a new BMS system for it.

Sources: DATA_RECONCILIATION_PR_SUMMARY.md

Summary

BMSview's data architecture is a robust system that balances client-side responsiveness with powerful serverless backend processing. Data flows through distinct, well-defined pathways depending on the task: rapid synchronous analysis for user uploads, flexible historical data aggregation for charting, a sophisticated ReAct loop for AI-powered insights, and an efficient pipeline for administrative data audits. The use of Netlify Functions, MongoDB, and a local-first approach provides a scalable and maintainable foundation for the application's features.