matteomorari/whatsapp-chat-analytics wiki

/matteomorari/whatsapp-chat-analytics

Wiki directory

Browse AI-generated documentation for any repository.

Relevant source files

The following files were used as context for generating this wiki page:

The WhatsApp Chat Analytics project provides a suite of tools to decrypt, search, and visualize WhatsApp chat history from Android backups. The entire process runs locally on the user's machine, ensuring that private messages are never uploaded to the internet. The system is composed of two primary parts: a Python-based decryption tool and a React-based web application for analysis.

This document outlines the architecture and workflow of the project, from decrypting the backup files to visualizing the chat data in the browser. The core principle is privacy and local processing; all data remains on the user's device throughout the analysis.

Sources: README.md:3-6, README.md:58-60

System Architecture

The project is divided into two main components: chatsExporter and web. The user first uses the Python script to decrypt their WhatsApp database, then uses the web application to upload and analyze the resulting data.

Sources: README.md:13-16

Overall Workflow

The following diagram illustrates the end-to-end user journey, from obtaining the encrypted backup to viewing analytics in the browser.

Generating architecture diagram...

This diagram shows the two main data inputs required by the web application: the decrypted msgstore.db and the model-storage-chat.json file.

Sources: README.md:26-57

Data Decryption (`chatsExporter`)

The chatsExporter/ directory contains a Python script responsible for decrypting .crypt15 WhatsApp backup files.

Sources: README.md:14, chatsExporter/whatsapp_db_android_exporter.py

Process Flow

The script whatsapp_db_android_exporter.py automates the decryption process by leveraging the wa-crypt-tools library. It reads configuration from a .env file, validates inputs, and executes command-line tools to produce a decrypted SQLite database.

Generating architecture diagram...

This diagram outlines the steps performed by the decryption script.

Sources: chatsExporter/whatsapp_db_android_exporter.py:70-130

Configuration

The script requires the following environment variables to be set in a .env file:

Variable	Description
`ENCRYPTION_KEY`	The 64-character hexadecimal key for the WhatsApp backup.
`TARGET_DIR`	The directory path containing the `.crypt15` backup files.

Sources: chatsExporter/whatsapp_db_android_exporter.py:7-9, chatsExporter/whatsapp_db_android_exporter.py:71-74

Web Application (`web/`)

The web/ directory contains a React single-page application that serves as the user interface for chat analysis. It runs entirely in the browser, using sql.js to query the uploaded database file locally.

Sources: README.md:15, web/package.json:15

Core Technologies

Technology	Purpose	Source
`react`	Building the user interface.	web/package.json:12
`sql.js`	Executing SQL queries on the database file directly in the browser.	web/package.json:15
`@amcharts/amcharts5`	Rendering interactive charts and visualizations.	web/package.json:5
`rsuite`	Providing UI components like pickers and uploaders.	web/package.json:14

User Workflow

The web application guides the user through a three-step process managed by the SearchSQLite component.

Generating architecture diagram...

This flow is controlled by the step state within the SearchSQLite component.

Sources: web/src/Pages/SearchSQLite/index.jsx:16, web/src/Pages/SearchSQLite/index.jsx:81-140

Step 1: File Upload

The UploadStep component provides drag-and-drop areas for the user to upload the msgstore.db and model-storage-chat.json files. These files are required to proceed.

msgstore.db: The decrypted SQLite database containing all messages.
model-storage-chat.json: A file containing a list of chats and their names, exported from WhatsApp Web.

Sources: web/src/Pages/SearchSQLite/components/UploadStep.jsx:14-17, README.md:53-55

Step 2: Search & Filter

The SearchStep component presents several filters to refine the messages to be analyzed.

Date Range: A DateRangePicker to select the start and end dates.
Media Type: A CheckPicker to include only specific message types (e.g., image, video, audio).
Message Direction: A SelectPicker for "Sent", "Received", or "All".
Chat: A SelectPicker to analyze a specific conversation or all chats.

Sources: web/src/Pages/SearchSQLite/components/SearchStep.jsx:29-88

Step 3: Results

The ResultsStep component displays the analysis of the filtered messages. It instantiates a MessageDataProcessor to generate data for the various display components.

ActivityChart: Visualizes message patterns (e.g., by hour of day, day of week).
MessagesOverTimeChart: Shows message volume trends over time (day, month, year).
StatisticsDisplay: Presents a table of key metrics about the chat.

Sources: web/src/Pages/SearchSQLite/components/ResultsStep.jsx:15-38

Data Analysis Pipeline

Once the user executes a search, the raw message data is processed through a pipeline to generate analytics. The MessageDataProcessor is the central orchestrator for this process.

Architecture

The MessageDataProcessor class takes the raw message array and uses specialized sub-processors to calculate different types of analytics. The results are cached to avoid re-computation.

Generating architecture diagram...

This diagram shows how MessageDataProcessor delegates tasks to specialized processors.

Sources: web/src/analytics/MessageDataProcessor.js:1-12, web/src/analytics/MessageDataProcessor.js:46, web/src/analytics/MessageDataProcessor.js:80, web/src/analytics/MessageDataProcessor.js:114, web/src/analytics/MessageDataProcessor.js:129

Statistics Generation

The StatisticsProcessor calculates a comprehensive set of metrics from the message list.

The following table details some of the key statistics generated:

Statistic	Description	Source
`total`	Total number of messages.	web/src/analytics/processors/statisticsProcessor.js:60
`firstMessageTime`	Timestamp of the first message in the set.	web/src/analytics/processors/statisticsProcessor.js:84
`lastMessageTime`	Timestamp of the last message in the set.	web/src/analytics/processors/statisticsProcessor.js:87
`longestAudio`	Duration of the longest audio message, in seconds.	web/src/analytics/processors/statisticsProcessor.js:79
`totalAudioDuration`	Sum of all audio message durations.	web/src/analytics/processors/statisticsProcessor.js:82
`avgAudioDuration`	Average duration of audio messages.	web/src/analytics/processors/statisticsProcessor.js:132
`longestDaysStreak`	The longest consecutive number of days with at least one message.	web/src/analytics/processors/statisticsProcessor.js:121
`dayWithMessages`	The total number of unique days that have at least one message.	web/src/analytics/processors/statisticsProcessor.js:108

These statistics are then rendered by the StatisticsDisplay component.

Sources: web/src/analytics/processors/statisticsProcessor.js, web/src/analytics/components/StatisticsDisplay.jsx:59-83

In-Browser Database Querying

The application queries the uploaded msgstore.db file without a server-side backend by using sql.js. The buildMessagesQuery utility function dynamically constructs an SQL query based on the user's filter selections.

Query Construction

The buildMessagesQuery function assembles a SELECT statement with a WHERE clause that incorporates conditions for the date range, selected chat, media types, and message direction.

// Example of a generated WHERE clause
const whereClause = whereConditions.join(' AND ');

// Example Query Snippet
`
SELECT
  'All Chats' AS chatName,
  mssg.from_me AS fromMe,
  Strftime('%Y-%m-%d %H:%M:%S', mssg.timestamp / 1000.0, 'unixepoch') AS dateTime,
  CASE mssg.message_type
    WHEN 0 THEN 'text'
    WHEN 1 THEN 'image'
    ...
  END AS type,
  ...
FROM message AS mssg
  ...
WHERE ${whereClause};
`

This approach allows for flexible and efficient data retrieval directly within the client's browser.

Sources: web/src/Pages/SearchSQLite/utils/queryBuilder.js:8-14, web/src/Pages/SearchSQLite/utils/queryBuilder.js:46-76

Home

System Architecture

Overall Workflow

Data Decryption (`chatsExporter`)

Process Flow

Configuration

Web Application (`web/`)

Core Technologies

User Workflow

Step 1: File Upload

Step 2: Search & Filter

Step 3: Results

Data Analysis Pipeline

Architecture

Statistics Generation

In-Browser Database Querying

Query Construction

Home

System Architecture

Overall Workflow

Data Decryption (`chatsExporter`)

Process Flow

Configuration

Web Application (`web/`)

Core Technologies

User Workflow

Step 1: File Upload

Step 2: Search & Filter

Step 3: Results

Data Analysis Pipeline

Architecture

Statistics Generation

In-Browser Database Querying

Query Construction

Home

System Architecture

Overall Workflow

Data Decryption (chatsExporter)

Process Flow

Configuration

Web Application (web/)

Core Technologies

User Workflow

Step 1: File Upload

Step 2: Search & Filter

Step 3: Results

Data Analysis Pipeline

Architecture

Statistics Generation

In-Browser Database Querying

Query Construction

Home

System Architecture

Overall Workflow

Data Decryption (chatsExporter)

Process Flow

Configuration

Web Application (web/)

Core Technologies

User Workflow

Step 1: File Upload

Step 2: Search & Filter

Step 3: Results

Data Analysis Pipeline

Architecture

Statistics Generation

In-Browser Database Querying

Query Construction

Data Decryption (`chatsExporter`)

Web Application (`web/`)

Data Decryption (`chatsExporter`)

Web Application (`web/`)