Data Crawler Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Data Crawler Engine is the core component of the Spotify API Data Collector system. Its primary purpose is to systematically browse the Spotify public API to collect metadata for musical tracks released between 2020 and 2024. It employs multi-stage discovery strategies to circumvent API limitations and ensures that only the most popular and data-rich entries are retained for analysis.
The engine operates as a modular pipeline: first discovering tracks via various search queries, then filtering them based on popularity metrics, and finally enriching the raw track data with detailed artist information such as genres and follower counts. The resulting data is persisted in a SQLite database and can be exported for external use.
Sources: README.md:3-12, app/main.py:28-40
Architecture and Discovery Strategies
The engine utilizes the SpotifyCrawler class to interface with Spotify's search endpoints. To maximize the variety and volume of tracks found for any given year, it implements three distinct search strategies.
Multi-Stage Search Logic
The discovery process is tiered to ensure the engine meets the configured TRACKS_PER_YEAR limit (defaulting to 1000):
- General Search: Uses a basic
year:YYYYquery to pull initial results. - Genre-Based Search: Iterates through a predefined list of genres (e.g., pop, rock, electronic) combined with the target year to find niche tracks.
- Keyword-Based Search: Uses popularity-linked keywords like "top", "hit", or "trending" to discover high-traffic content.
Sources: service/crawler.py:24-48, app/config.py:17
Data Flow Diagram
The following diagram illustrates the lifecycle of a data collection cycle from initial trigger to final storage.
Core Components
SpotifyCrawler Class
Located in service/crawler.py, this class manages the interaction logic with the spotipy library. It handles pagination through search results in batches of 50 and implements rate limiting to respect Spotify's API constraints.
| Function | Description |
|---|---|
crawl_tracks_by_year | Orchestrates the three search strategies for a specific year. |
_search_with_query | Handles the low-level pagination and deduplication logic using a seen_ids set. |
filter_top_popular_tracks | Sorts discovered tracks by popularity and truncates the list to TOP_N_TRACKS. |
enrich_track_with_artist | Makes additional API calls to fetch full artist profiles for specific track IDs. |
Sources: service/crawler.py:13-132
DataProcessor Class
The DataProcessor acts as the orchestrator between the crawler and the storage layer. It is responsible for calling the crawler's methods, managing the SQLite database connection via app/database.py, and handling the transformation of raw API responses into Track objects.
Data Model and Configuration
Track Data Structure
The engine transforms disparate API responses into a unified Track dataclass defined in app/track.py. This model combines track-level data with artist-level metadata.
| Field | Type | Source |
|---|---|---|
track_id | str | Spotify Track Object id |
track_popularity | int | Spotify Track Object popularity (0-100) |
artist_followers | int | Spotify Artist Object followers.total |
artist_genres | str | Comma-separated list from Artist genres |
duration_ms | int | Track length in milliseconds |
Sources: app/track.py:7-25, README.md:23-38
Configuration Parameters
The engine's behavior is governed by environment variables managed through the Config class.
| Constant | Default Value | Description |
|---|---|---|
TRACKS_PER_YEAR | 1000 | Max tracks to discover per year during search. |
TOP_N_TRACKS | 200 | Number of tracks to keep after popularity filtering. |
API_RATE_LIMIT_DELAY | 0.1 | Seconds to wait between search requests. |
ARTIST_FETCH_DELAY | 0.05 | Seconds to wait between artist enrichment calls. |
Sources: app/config.py:11-23
Summary
The Data Crawler Engine provides a robust automated pipeline for musical data acquisition. By combining multiple search heuristics with a strict popularity filter and detailed metadata enrichment, it ensures a high-quality dataset. Its integration with SQLite and CSV export functionality makes it a complete solution for long-term music trend analysis.
Sources: README.md:5-20, app/main.py:53-60