Deduplication Logic
Relevant source files
The following files were used as context for generating this wiki page:
- app/src/main/kotlin/com/shadowcheck/mobile/utils/DeduplicationUtil.kt
- app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt
- app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/ui/screens/SettingsScreen.kt
- app/src/main/kotlin/com/shadowcheck/mobile/ui/screens/settings/DatabaseStats.kt
- app/src/main/kotlin/com/shadowcheck/mobile/ui/screens/StatsScreen.kt
Introduction
The deduplication logic in ShadowCheck Mobile is a critical system designed to maintain database integrity and optimize storage efficiency. Its primary purpose is to ensure that the application handles the high volume of signals detected during wardriving—including WiFi networks, Bluetooth/BLE devices, and cellular towers—without creating redundant or stale records.
This system operates at two levels: real-time deduplication during the scanning process to prevent immediate redundancy, and batch deduplication via utility classes for database maintenance. By keeping only the latest or most relevant sightings based on unique identifiers (like BSSID, MAC address, or Cell ID), the logic ensures that the network map and statistical analytics remain accurate and performant.
Sources: app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt, app/src/main/kotlin/com/shadowcheck/mobile/utils/DeduplicationUtil.kt
Real-time Scanning Deduplication
The CompleteScannerService implements a "30-second deduplication" window. When a signal is detected, the service checks the ShadowCheckDatabase for an existing record associated with the unique hardware identifier. A new database insertion or update only occurs if the device is new or if at least 30,000 milliseconds have elapsed since the last recorded sighting.
Data Flow for Live Scans
This diagram illustrates the logic applied during active network scanning to filter incoming signals before they reach the persistent storage layer.
The logic differentiates between "unique" sightings (the first time a hardware ID is seen) and "total" observations (every time a record is updated after the 30-second window).
Sources: app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt:50-70, app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt:110-125, app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt:185-200
Batch Database Deduplication
The DeduplicationUtil object provides a mechanism for manual or scheduled maintenance of the database. It processes the entire collection of detected networks to prune duplicates that may have accumulated over time or through data imports (such as WiGLE CSV imports).
Components of Deduplication Utility
The utility returns a DeduplicationResult containing the count of removed items for each category.
| Component | Description | Logic Identifier |
|---|---|---|
| WiFi Deduplication | Groups networks by BSSID; keeps the record with the latest timestamp. | wifiRemoved |
| Bluetooth Deduplication | Groups devices by hardware address; keeps the latest sighting. | btRemoved |
| Cellular Deduplication | Groups towers by Cell ID; keeps the most recent data. | cellRemoved |
Sources: app/src/main/kotlin/com/shadowcheck/mobile/utils/DeduplicationUtil.kt:7-15
Maintenance Flow
Users can trigger this logic through the application settings. The following sequence shows the interaction between the UI and the deduplication utility.
Database Integrity Maintenance
The application provides additional tools to clean the database that supplement standard deduplication:
- GPS Filtering: A specific cleanup function allows users to delete records that lack GPS coordinates, ensuring the quality of mapped data.
- Sensor Purging: Users can purge the
SensorReadingtable independently of the network sighting tables. - Vacuum/Optimization: Beyond record-level deduplication, the system supports a
VACUUM(Optimize Database) command to reclaim storage space from deleted duplicate records.
Sources: app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/ui/screens/SettingsScreen.kt:125-155, app/src/main/kotlin/com/shadowcheck/mobile/ui/screens/settings/DatabaseStats.kt:123-130
Statistics and Reporting
The results of deduplication are reflected in the application's statistics screens, which distinguish between unique hardware assets and total sighting events.
| Metric | Derivation Logic | Source File |
|---|---|---|
| Unique Count | dao.getUniqueCount() - Count of distinct IDs. | StatsScreen.kt |
| Total Count | dao.getTotalCount() - Count of all observations. | StatsScreen.kt |
| Duplicates Removed | Result of the batch deduplication utility run. | DeduplicationUtil.kt |
Sources: app/src/main/kotlin/com/shadowcheck/mobile/ui/screens/StatsScreen.kt:35-45, app/src/main/kotlin/com/shadowcheck/mobile/utils/DeduplicationUtil.kt:7-11
Conclusion
The deduplication logic in ShadowCheck Mobile serves as a multi-tier filter. By combining a 30-second real-time window with manual batch cleaning tools, the system balances the need for historical sighting data (tracking device movement or recurrence) with the need for a clean, optimized database that represents the unique physical landscape of signals detected by the hardware.
Sources: app/src/main/kotlin/com/shadowcheck/mobile/rebuilt/service/CompleteScannerService.kt, app/src/main/kotlin/com/shadowcheck/mobile/utils/DeduplicationUtil.kt