Application Deployment

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• app/api.py
• app/config.py
• app/main.py
• scripts/export_csv.py
• service/data_processor.py

The Spotify API Data Collector deployment process involves setting up a local Python environment, configuring environment variables for Spotify API access, initializing a SQLite database, and running two distinct components: a data collection crawler and a FastAPI web server. The system is designed with a modular architecture to facilitate data extraction from the Spotify Web API and subsequent exposure via RESTful endpoints.

Sources: README.md:1-18, app/main.py:1-15

Environment Setup and Configuration

The application requires a Python 3.11 environment. Deployment begins with virtual environment creation and dependency installation via pip. Configuration is managed through a .env file located at the root of the project, which stores sensitive credentials and operational parameters.

Required Environment Variables

Variable	Type	Description	Default
SPOTIPY_CLIENT_ID	String	Spotify Developer Client ID	Required
SPOTIPY_CLIENT_SECRET	String	Spotify Developer Client Secret	Required
DATABASE_URL	String	SQLAlchemy-compatible database URL	sqlite:///spotify_data.db
TRACKS_PER_YEAR	Integer	Total tracks to search per year	1000
TOP_N_TRACKS	Integer	Number of top tracks to retain after filtering	200
API_RATE_LIMIT_DELAY	Float	Delay between API calls (seconds)	0.1

Sources: README.md:65-88, app/config.py:11-23

The configuration is validated at runtime using the Config.validate() method, which ensures that mandatory Spotify credentials are present before any service initializes.

Sources: app/config.py:27-35, app/main.py:24-30

Data Collection Pipeline

The deployment of the data collector is executed by running the app/main.py module. This orchestrates the initialization of the SpotifyClient, SpotifyCrawler, and DataProcessor.

Collection Flow

The process follows a sequential flow:

1. Search: Querying Spotify for tracks within the range 2020-2024.
2. Filter: Selecting the top popular tracks (defaulting to the top 200).
3. Enrich: Fetching detailed artist metadata (followers, genres).
4. Persist: Saving data to the SQLite database.

The data processor logs progress every 20 tracks during the enrichment phase to provide feedback during deployment execution.

Sources: app/main.py:33-47, service/data_processor.py:44-78

API Service Deployment

The web API is deployed using uvicorn, serving the FastAPI application defined in app/api.py. The service automatically handles database initialization upon startup.

API Execution Command

python3.11 -m uvicorn app.api:app --host 0.0.0.0 --port 8000 --reload

The application lifecycle includes a startup event that ensures the database schema is present before accepting requests.

Data Export and Utility Scripts

Beyond the core services, deployment includes utility scripts for data management and portability.

CSV Export

Users can manually trigger a data export from the persisted SQLite database to a CSV file using the scripts/export_csv.py script. This script supports a custom output path via the -o flag.

Final Cleaning

A specific cleaning script, scripts/Clean_finale.py, is used to refine the dataset by:

• Removing "Test Artist" or "Test" entries.
• Limiting the dataset to exactly 200 tracks per year based on popularity.
• Generating the final spotify_data_final.csv.

Sources: scripts/export_csv.py:34-55, scripts/Clean_finale.py:10-30

Service Monitoring

Monitoring is facilitated through integrated logging and a health check endpoint.

• Logs: Operation logs are stored in the logs/ directory.
• Health Endpoint: GET /health provides a JSON response {"status": "healthy", "service": "spotify-api"} to verify service availability.

Sources: app/main.py:58, app/api.py:202-205

The deployment is considered successful when the API documentation is accessible at http://127.0.0.1:8000/docs, confirming that the FastAPI application is correctly mapping endpoints to the underlying SQLite data store.

Sources: README.md:105-108