Idempotency & Conflict Resolution
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
In the Spotify API Data Collector, idempotency and conflict resolution are critical for maintaining data integrity during long-running collection processes. The system is designed to allow multiple executions of the crawler without creating duplicate entries or corrupting the existing dataset. This is primarily achieved through unique identifiers and strategic filtering at both the application and database layers.
The scope of these mechanisms covers the lifecycle of a "track" object—from its discovery via the Spotify Web API to its final storage in the SQLite database and eventual cleanup. By ensuring idempotency on the track_id, the system guarantees that re-running the collection (e.g., after a crash or for updates) remains safe and efficient.
Data Identification & Uniqueness
The foundation of the project's conflict resolution strategy is the use of the Spotify-provided track_id as a Primary Key. This unique identifier ensures that every record in the system corresponds to exactly one specific track on the Spotify platform.
Unique Constraints
The database schema defines track_id as the Primary Key (PK) for the tracks table. This prevents the database from accepting two rows with the same identifier, effectively neutralizing potential duplicates at the storage level.
Sources: README.md:31, test/test_database.py:61-68
| Field | Type | Constraint | Description |
|---|---|---|---|
track_id | str | PRIMARY KEY | Unique Spotify ID used for deduplication |
year | int | INDEX | Indexed for performance, not uniqueness |
artist_name | str | INDEX | Indexed for performance |
Sources: README.md:31-45
Application-Level Deduplication
The SpotifyCrawler implements a local tracking mechanism to avoid redundant processing of the same track within a single execution cycle.
Tracking Seen Identifiers
During the crawling process, a Python set named seen_ids is used to keep track of every track_id encountered across different search strategies (general search, genre-based, and keyword-based). Before adding a track to the results list, the crawler checks this set.
The diagram shows the logic within _search_with_query where tracks are filtered before being returned to the main crawler loop.
Sources: service/crawler.py:46-77
Database Conflict Resolution
When the DataProcessor attempts to save batches of tracks to the database, it utilizes specific SQL patterns to handle potential conflicts where a track might already exist from a previous run.
Insertion Logic
The system supports both individual track insertion and batch processing. The batch insertion mechanism specifically handles the logic of committing multiple records while respecting the unique constraints of the database.
Testing for Idempotency
Unit tests confirm that inserting the same track twice does not result in a duplicate entry or a crash. The test suite validates that the database correctly ignores or handles the second insertion attempt, maintaining a single record.
Sources: test/test_database.py:86-94
Post-Collection Cleanup
Conflict resolution also extends to data cleaning through specialized scripts. These scripts resolve "conflicts" between test data and production data, as well as managing data volume constraints.
Test Data Removal
The Clean_finale.py script and DataProcessor.clean_test_data method identify and remove tracks that match test patterns (e.g., artist name containing "Test Artist" or track name starting with "Test"). This ensures that development artifacts do not pollute the final analysis dataset.
Sources: scripts/Clean_finale.py:15-19, service/data_processor.py:157-183
Popularity-Based Conflict Resolution
When more than the desired number of tracks are collected for a specific year (due to overlapping search results), the system resolves this by keeping only the most popular tracks.
- Crawler Level: Uses
filter_top_popular_tracksto sort by popularity and slice the top N. - Script Level:
Clean_finale.pyperforms a globalnlargest(200, 'track_popularity')per year to ensure only the highest-quality data remains.
Sources: service/crawler.py:108-112, scripts/Clean_finale.py:22-27
Summary
The Spotify API Data Collector ensures idempotency primarily through the strict enforcement of track_id uniqueness. By combining memory-based sets (seen_ids) during crawling, Primary Key constraints in SQLite, and popularity-based filtering during post-processing, the system maintains a clean, duplicate-free dataset even across multiple interrupted or redundant execution cycles.