Project File Structure
Relevant source files
The following files were used as context for generating this wiki page:
The Spotify API Data Collector is a modular system designed to fetch music metadata from Spotify, persist it in a local SQLite database, and expose it via a RESTful API. The project architecture is divided into clear functional layers: application logic (app/), core services for external communication and processing (service/), and utility scripts (scripts/).
Sources: README.md:1-15, app/main.py:1-10
Core Application Layer (app/)
This directory contains the entry points for the system, the data models, and the configuration management.
Application Components
- main.py: The primary entry point for the data collection pipeline. It orchestrates the initialization of services and the execution of the crawler.
- api.py: Implementation of the FastAPI web server, providing endpoints for querying tracks, statistics, and health checks.
- config.py: Centralized configuration management using environment variables for API credentials and operational parameters.
- track.py: Defines the
Trackdataclass, which serves as the primary data structure for metadata throughout the system.
Sources: app/main.py:12-60, app/api.py:1-25, app/config.py:1-30, app/track.py:1-50
Data Model Structure
The system centers around the Track model, which encapsulates both Spotify track details and enriched artist information.
| Field | Type | Description |
|---|---|---|
track_id | str | Unique Spotify ID (Primary Key) |
track_name | str | Name of the track |
track_popularity | int | Popularity score (0-100) |
album_name | str | Name of the album |
year | int | Release year (indexed) |
artist_name | str | Name of the artist (indexed) |
artist_genres | str | Comma-separated list of genres |
Sources: README.md:27-41, app/track.py:8-21
Service Layer (service/)
The service/ directory contains the business logic for interacting with external APIs and processing data.
Service Modules
- spotify_client.py: A wrapper around the
spotipylibrary that handles authentication (Client Credentials flow) and low-level API requests. - crawler.py: Implements the logic for searching and discovering tracks. It uses multiple strategies (general search, genres, and keywords) to find tracks by year.
- data_processor.py: Orchestrates the flow between the crawler and the database. It handles batch insertions and prepares data for CSV export.
Sources: service/spotify_client.py:15-60, service/crawler.py:18-45, service/data_processor.py:46-75
Data Flow Diagram
The following diagram illustrates how data flows from the Spotify API through the services to the database.
The DataProcessor acts as the intermediary, ensuring that tracks gathered by the SpotifyCrawler are properly enriched and stored.
Sources: service/data_processor.py:46-75, app/main.py:38-50
Web API and Scripts
The project provides access to the data through a web interface and specialized scripts.
API Endpoints
FastAPI is used to expose the collected data.
Sources: app/api.py:53-150, README.md:88-120
Utility Scripts (scripts/)
Scripts are provided for administrative and export tasks:
- export_csv.py: A standalone script to dump the contents of the SQLite database into a
spotify_data.csvfile. - Clean_finale.py: A cleaning script that removes test data and limits the dataset to the top 200 tracks per year based on popularity.
Sources: scripts/export_csv.py:25-50, scripts/Clean_finale.py:10-35
System Configuration
The application relies on environment variables defined in a .env file, managed via the Config class.
| Variable | Default | Description |
|---|---|---|
SPOTIPY_CLIENT_ID | Required | Spotify Developer Client ID |
SPOTIPY_CLIENT_SECRET | Required | Spotify Developer Client Secret |
DATABASE_URL | sqlite:///spotify_data.db | Connection string for SQLite |
TRACKS_PER_YEAR | 1000 | Target number of tracks to fetch per year |
TOP_N_TRACKS | 200 | Number of top tracks to keep after filtering |
Sources: app/config.py:11-28, README.md:65-75
Conclusion
The project's file structure supports a clear separation of concerns, where the app/ directory handles the interface and models, the service/ directory manages external data acquisition and internal processing, and the scripts/ directory provides maintenance tools. This modularity allows for independent scaling of the crawler logic and the API delivery.