Data Flow
Relevant source files
The following files were used as context for generating this wiki page:
- web/src/Pages/SearchSQLite/hooks/useFileUpload.js
- web/src/analytics/MessageDataProcessor.js
- chatsExporter/whatsapp_db_android_exporter.py
- web/src/Pages/SearchSQLite/index.jsx
- web/src/Pages/SearchSQLite/components/ResultsStep.jsx
- web/src/analytics/processors/statisticsProcessor.js
- web/src/analytics/processors/timePeriodProcessor.js
- README.md
The data flow in whatsapp-chat-analytics is designed to be entirely client-side, ensuring user privacy by processing all data within the user's browser. The flow begins with the decryption of WhatsApp's Android backup files and culminates in the rendering of interactive charts and statistics in a web application.
The process is divided into two main parts: an offline data preparation stage using a Python script, and an in-browser analysis stage powered by a React application. This document outlines the journey of the data from its encrypted source to its final visual representation.
High-Level Data Flow
The overall data flow can be visualized as a multi-stage process, starting with user-provided files and ending with browser-based analytics.
This diagram shows the two main phases: preparing the data files offline and then uploading them to the web application for analysis.
Sources: README.md, chatsExporter/whatsapp_db_android_exporter.py, web/src/Pages/SearchSQLite/index.jsx
Stage 1: Data Decryption and Preparation
Before any analysis can occur in the web application, the user must prepare two key data files.
Decrypting the Message Database
The primary data source is the encrypted WhatsApp message backup, typically a .crypt15 file. The Python script whatsapp_db_android_exporter.py is responsible for decrypting this file.
- Input: The script requires the user's 64-character
ENCRYPTION_KEYand the path to the directory containing the.crypt15files, configured in a.envfile. - Process: It uses the
wa-crypt-toolslibrary, specifically thewadecryptcommand-line tool, to decrypt the backup. - Output: The script produces a decrypted SQLite database file,
msgstore.db.
This msgstore.db file contains all message history, media references, and timestamps in a readable format.
Sources: chatsExporter/whatsapp_db_android_exporter.py:100-107, README.md:54-63
Exporting the Chat List
To map chat IDs to user/group names, a second file is required. This is generated by running a JavaScript snippet in the WhatsApp Web developer console.
- Process: The user copies the content of
dumpWAChats.jsand runs it as a snippet in their browser's developer tools while on WhatsApp Web. - Output: This action downloads a
model-storage-chat.jsonfile, which contains a list of all chats, their names, and their IDs.
Sources: README.md:65-79
Stage 2: Client-Side Data Ingestion
Once the msgstore.db and model-storage-chat.json files are prepared, the user uploads them into the web application. This process is managed by the SearchSQLite page.
This sequence shows how the useFileUpload hook orchestrates reading and parsing the uploaded files into usable data structures.
Sources: web/src/Pages/SearchSQLite/hooks/useFileUpload.js:23-32, web/src/Pages/SearchSQLite/components/UploadStep.jsx:20-26
The useFileUpload hook is a custom hook responsible for handling the file upload logic. It differentiates between the 'messages' database and the 'chats' JSON file, using the appropriate service to process each one.
msgstore.db: Read as anArrayBufferand initialized byDatabaseServicewhich likely loads it intosql.js.model-storage-chat.json: Read as text and parsed into a JavaScript array byFileService.
Sources: web/src/Pages/SearchSQLite/hooks/useFileUpload.js:18-28
Stage 3: Data Processing and Analysis
After the files are uploaded and initially processed, the raw message data is passed to the MessageDataProcessor for in-depth analysis. This class acts as a central orchestrator, delegating specific processing tasks to specialized sub-processors.
The MessageDataProcessor Facade
The MessageDataProcessor is instantiated with the list of messages retrieved from the database. It serves as a single point of entry for the UI components to request analytical data. It uses lazy-loading and caching to avoid re-computing data. For example, getMessagesPerDay() will only compute the data on its first call and return the cached result on subsequent calls.
Sources: web/src/analytics/MessageDataProcessor.js:11-30, web/src/Pages/SearchSQLite/components/ResultsStep.jsx:13
Specialized Processors
The MessageDataProcessor delegates tasks to a suite of processors, each with a distinct responsibility.
This diagram illustrates the relationship where MessageDataProcessor acts as a facade, invoking methods on specialized processors to generate different analytical views.
Sources: web/src/analytics/MessageDataProcessor.js:1-4
| Processor Class | Responsibility | Key Methods | Source File |
|---|---|---|---|
TimePeriodProcessor | Groups messages by day, month, or year. | processMessagesByDay() | web/src/analytics/processors/timePeriodProcessor.js |
PatternProcessor | Groups messages by recurring time patterns (e.g., hour of day). | processMessagesByHour() | web/src/analytics/processors/patternProcessor.js |
HeatmapProcessor | Generates data structures suitable for heatmap visualizations. | processHeatmapHourPerDay() | web/src/analytics/processors/heatmapProcessor.js |
StatisticsProcessor | Calculates aggregate statistics like totals, streaks, and averages. | processStatistics() | web/src/analytics/processors/statisticsProcessor.js |
Stage 4: Data Visualization
The final stage in the data flow is rendering the processed analytics data. The ResultsStep component is the primary consumer of the MessageDataProcessor.
- It instantiates
MessageDataProcessorwith the filtered message list. - It calls the various
get...()methods on the processor to retrieve the data needed for each visualization component. - It passes this data as props to child components like
ActivityChart,MessagesOverTimeChart, andStatisticsDisplay.
// web/src/Pages/SearchSQLite/components/ResultsStep.jsx:13-15
const dataProcessor = new MessageDataProcessor(messagesList, startDateSelected, endDateSelected);
const chatName = dataProcessor.chatNames[0];
const stats = dataProcessor.getMessagesStatistics()[chatName];
These child components are responsible for rendering the data using the amCharts library or custom React components. For example, ActivityChart contains a SelectPicker that allows the user to switch between different views (e.g., "Hour per day", "Heat Map"), which in turn calls the corresponding method on the dataProcessor instance to get the correct data for the selected chart.
Sources: web/src/Pages/SearchSQLite/components/ResultsStep.jsx:17-33, web/src/analytics/components/ActivityChart.jsx:16-34
Conclusion
The data flow of whatsapp-chat-analytics is a well-defined, sequential process that prioritizes user privacy by keeping all sensitive data and processing logic within the browser. It starts with offline decryption, moves to in-browser file ingestion and parsing, proceeds to a modular analysis pipeline orchestrated by MessageDataProcessor, and concludes with the visualization of rich analytics. This architecture effectively separates data preparation, processing, and presentation concerns.