Project Overview

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• app/api.py
• app/database.py
• app/main.py
• service/crawler.py
• service/spotify_client.py
• app/config.py
• app/track.py

The Spotify API Data Collector is a comprehensive system designed to harvest track metadata from the Spotify Public API and expose it through a FastAPI web interface. The primary objective is to collect, filter, and analyze music data from the years 2020 to 2024, persisting this information in a local SQLite database for further consumption or export.

Sources: README.md:3-8, app/main.py:12-16

System Architecture

The project is built using a modular architecture that separates data acquisition, processing, storage, and presentation.

Component Map

The following diagram illustrates the high-level flow from the external Spotify API to the end-user interfaces.

Data Acquisition (Crawler & Client)

The acquisition layer uses a multi-strategy approach to ensure a high volume of relevant tracks is collected for each target year.

Crawler Strategies

The SpotifyCrawler implements three distinct strategies to find tracks:

1. General Search: Uses the year:YYYY query filter.
2. Genre-based Search: Iterates through a list of 18 genres (e.g., pop, rock, hip hop) combined with the target year.
3. Keyword-based Search: Uses popularity-centric keywords (e.g., "top", "hit", "trending").

Sources: service/crawler.py:25-50

Rate Limiting and Enrichment

The SpotifyClient manages the connection to Spotify using the Client Credentials flow. To remain compliant with API limits, the system implements mandatory delays:

• API_RATE_LIMIT_DELAY: 0.1s (default) after search requests.
• ARTIST_FETCH_DELAY: 0.05s (default) after artist metadata requests.

Sources: app/config.py:20-21, service/spotify_client.py:48-61

Data Model and Storage

Data is modeled using a structured Track class and stored in an indexed SQLite database.

Track Entity Schema

Field	Type	Description
track_id	str (PK)	Unique Spotify ID (Primary Key)
track_name	str	Name of the track
track_popularity	int	Popularity score (0-100)
album_name	str	Name of the containing album
year	int	Release year (Indexed)
artist_name	str	Name of the primary artist (Indexed)
artist_genres	str	Comma-separated list of genres

Sources: README.md:23-38, app/track.py:8-22, app/database.py:12-30

Database Initialization Flow

Web API Interface

The system provides a FastAPI-based REST interface for querying the collected data.

API Endpoints

Endpoint	Method	Description
/tracks	GET	List tracks with pagination and filtering (year, min_popularity)
/tracks/year/{year}	GET	Fetch tracks specifically for one year (2020-2024)
/tracks/artist/{name}	GET	Case-insensitive search for tracks by artist name
/stats	GET	Global statistics including top artists and average popularity
/health	GET	Service status check

Sources: app/api.py:41-180, README.md:88-105

Configuration and Execution

The project relies on a .env file for Spotify credentials and operational parameters.

Configuration Constants

# app/config.py
TRACKS_PER_YEAR = 1000 # Max tracks to fetch per year
TOP_N_TRACKS = 200    # Number of top tracks to retain after filtering
YEARS = [2020, 2021, 2022, 2023, 2024]

Sources: app/config.py:15-18

Lifecycle of a Collection Run

1. Validation: Config.validate() checks for required Spotify keys.
2. Collection: processor.collect_data_for_years() triggers the crawler.
3. Filtering: Tracks are sorted by popularity; only the top N are kept.
4. Enrichment: Artist details (followers, genres) are fetched for filtered tracks.
5. Persistence: Data is saved to spotify_data.db using batch inserts.
6. Export: A final spotify_data.csv is generated.

Sources: app/main.py:19-50, service/crawler.py:108-115

Conclusion

The Spotify API Data Collector provides a robust pipeline for music data analysis. By combining multi-strategy crawling with automated SQLite indexing and a modern FastAPI frontend, it offers both researchers and developers a streamlined tool for exploring Spotify's historical track performance data.