Crawler & Spotify Client Tests
Relevant source files
The following files were used as context for generating this wiki page:
The test suite for the Spotify API Data Collector focuses on validating two primary service components: the SpotifyClient, which manages low-level API communications and authentication, and the SpotifyCrawler, which implements high-level data collection strategies. These tests ensure that the system correctly interacts with the Spotify Web API, handles rate limiting, processes paginated search results, and maintains data integrity during the enrichment process.
The testing infrastructure utilizes pytest along with unittest.mock to simulate API responses, allowing for comprehensive verification of error handling, data filtering logic, and edge cases (such as missing artist data or empty search results) without making actual network requests to Spotify.
Sources: test/test_crawler_extended.py, test/test_spotify_client_extended.py, service/crawler.py:1-20
Spotify Client Validation
The SpotifyClient tests verify the core communication layer between the application and the Spotify Web API. This component is responsible for executing the Client Credentials flow for authentication and performing specific queries for tracks and artists.
Core Client Functionality
Tests ensure that the client correctly initializes using environment variables and handles various API response scenarios. Key methods validated include search_tracks_by_year, get_artist, and get_track.
The sequence above illustrates the verification process for artist data retrieval, ensuring exceptions are correctly propagated when the API fails.
Sources: test/test_spotify_client_extended.py:15-60, service/spotify_client.py:48-68
Client Test Categories
| Test Case | Description | Source File |
|---|---|---|
test_init | Validates that the client initializes and authenticates via SpotifyClientCredentials. | test/test_spotify_client.py:16-20 |
test_search_tracks_by_year | Verifies the search query construction and result parsing for a specific year. | test/test_spotify_client.py:24-43 |
test_get_artist_success | Confirms successful retrieval of artist metadata including followers and genres. | test/test_spotify_client_extended.py:15-38 |
test_get_track | Validates retrieval of specific track details by Spotify ID. | test/test_spotify_client_extended.py:64-83 |
Crawler Logic and Strategy
The SpotifyCrawler tests focus on the high-level orchestration of data collection. The crawler employs multiple strategies (General search, Genre-based, and Keyword-based) to reach target track counts per year.
Search and Pagination Strategies
The crawler must handle Spotify's 50-item limit per request by paginating through offsets. Tests simulate these multiple "pages" to ensure the crawler continues collecting until the limit is reached or the API is exhausted.
This flow represents the logic verified in pagination tests where the mock returns sequential result sets until an empty list is encountered.
Sources: test/test_crawler_extended.py:30-60, service/crawler.py:46-83
Popularity Filtering
A critical feature of the crawler is the ability to filter collected data down to the "Top N" most popular tracks. Tests verify that the sorting logic correctly prioritizes the popularity field (0-100 scale).
def test_filter_top_popular_tracks(self):
# Verification that only the highest popularity scores are kept
tracks = [
{'id': '1', 'popularity': 30},
{'id': '2', 'popularity': 80},
{'id': '3', 'popularity': 50},
{'id': '4', 'popularity': 90},
]
top_tracks = crawler.filter_top_popular_tracks(tracks, top_n=2)
assert top_tracks[0]['popularity'] == 90 # Source: [test/test_crawler.py:17-31]
Sources: test/test_crawler.py:17-31, service/crawler.py:121-126
Data Enrichment and Model Integration
The enrich_track_with_artist method bridges raw API data and the structured Track model. Testing ensures that track data is merged with artist details (genres, followers) and converted into a validated Track object.
Enrichment Flow
Tests validate that the crawler can handle partial failures, such as when track data is present but the artist metadata request fails.
| Scenario | Expected Behavior | Source |
|---|---|---|
| Successful Enrichment | Returns a Track object with full artist/album metadata. | test/test_crawler_extended.py:141-164 |
| API Error on Artist | Returns Track with default values (0 followers) for artist fields. | test/test_crawler_extended.py:166-185 |
| No Artist in Data | Raises ValueError: Track sans artiste. | test/test_crawler_extended.py:187-196 |
The relationship verified in tests shows the Crawler acting as a factory for the Track model using external API data.
Sources: service/crawler.py:128-157, app/track.py:35-65
Conclusion
The Crawler and Spotify Client tests provide a robust verification layer for the application's data acquisition engine. By mocking external dependencies, the suite confirms that search strategies, rate-limit delays, and data transformation logic function correctly, ensuring that only high-quality, enriched musical data enters the system's database.