Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• app/main.py
• app/api.py
• app/config.py
• service/crawler.py
• service/spotify_client.py
• service/data_processor.py
• README.md

The Spotify API Data Collector is a modular system designed to collect, enrich, store, and expose music track metadata from the Spotify Web API. The system focuses on retrieving popular tracks from specific years (2020-2024), processing them with detailed artist information, and providing access through both a FastAPI web interface and CSV exports.

Sources: README.md:3-13, app/main.py:11-20

Core System Components

The architecture is divided into three primary layers: the Data Acquisition Layer (Crawler and Client), the Persistence Layer (SQLite), and the Presentation Layer (FastAPI and Scripts).

Component Summary

Component	Responsibility	Key Files
SpotifyClient	Low-level API interaction and authentication management.	service/spotify_client.py
SpotifyCrawler	Multi-strategy search logic and track enrichment.	service/crawler.py
DataProcessor	Orchestrates collection workflows and data exports.	service/data_processor.py
FastAPI App	RESTful interface for querying stored data.	app/api.py
SQLite DB	Persistent storage with indexed fields for performance.	app/config.py, README.md

Sources: README.md:15-22, app/main.py:28-36

Data Acquisition Flow

The data collection process is managed by DataProcessor, which utilizes the SpotifyCrawler to execute searches and the SpotifyClient to handle the underlying HTTP requests to Spotify.

The system employs multiple strategies to find tracks: general year-based queries, genre-specific searches, and keyword-based searches (e.g., "top", "hit").

Sources: app/main.py:23-53, service/crawler.py:27-51

Spotify API Interaction

The SpotifyClient uses the SpotifyClientCredentials flow for authentication. It implements rate limiting by introducing delays between requests to ensure compliance with API quotas.

def search_tracks_by_year(self, year: int, limit: int = 1000, offset: int = 0):
    # ... logic for query construction ...
    results = self.sp.search(q=query, type='track', limit=min(limit, 50), offset=offset)
    time.sleep(Config.API_RATE_LIMIT_DELAY)
    return results

Sources: service/spotify_client.py:27-46, app/config.py:20-21

Data Model and Persistence

Data is stored in a SQLite database. The schema is designed for quick retrieval by indexing fields commonly used in filters, such as year, artist_name, and track_popularity.

Database Schema (Table: tracks)

Field	Type	Description
track_id	str (PK)	Unique Spotify identifier.
track_name	str	Name of the music track.
track_popularity	int	Popularity score (0-100).
artist_name	str	Name of the primary artist (Indexed).
year	int	Release year (Indexed).
artist_genres	str	Comma-separated list of genres.
duration_ms	int	Track duration in milliseconds.

Sources: README.md:27-44

API Architecture

The Presentation Layer is built using FastAPI. It provides endpoints for filtering tracks by year, artist, or popularity, and computes global statistics on the collected dataset.

System Configuration

The application is configured via environment variables, which are validated at startup to ensure the required Spotify credentials (SPOTIPY_CLIENT_ID, SPOTIPY_CLIENT_SECRET) are present.

Option	Default	Description
TRACKS_PER_YEAR	1000	Max tracks to crawl per year.
TOP_N_TRACKS	200	Number of tracks to keep after popularity filtering.
API_RATE_LIMIT_DELAY	0.1	Seconds to wait between standard API calls.
DATABASE_URL	sqlite:///spotify_data.db	Path to the SQLite storage file.

Sources: app/config.py:12-25, README.md:65-74