Data Processing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Data Processing Pipeline is a centralized system designed to orchestrate the collection, enrichment, storage, and export of music track metadata from the Spotify API. It acts as the intermediary layer between the raw API data fetched by the crawler and the persistent storage or analytical outputs required by the application.
The pipeline operates by iterating through a defined set of years, collecting high-popularity tracks, enriching them with detailed artist information, and ensuring data integrity through batch database insertions and idempotent operations. It also provides utilities for exporting processed data to CSV format for external analysis.
Pipeline Architecture and Orchestration
The pipeline is primarily managed by the DataProcessor class, which integrates the SpotifyCrawler for data acquisition and the Database module for persistence.
The workflow begins with configuration validation to ensure Spotify API credentials are present, followed by a looped process for each targeted year.
Sources: app/main.py:16-45, service/data_processor.py:28-56
Key Components
| Component | Description |
|---|---|
DataProcessor | Orchestrates the flow between crawling, database operations, and CSV exports. |
SpotifyCrawler | Handles complex search strategies (general, genre-based, keyword-based) and artist enrichment. |
Track | A data model representing enriched track information, including artist followers and genres. |
SQLite Database | Persistent storage with indexed fields for year, artist_name, and track_popularity. |
Sources: service/data_processor.py:17-26, service/crawler.py:17-45, app/track.py:7-22, README.md:38-51
Data Collection and Enrichment Logic
The collection phase utilizes multiple strategies to maximize the diversity and volume of tracks retrieved for each year.
Search Strategies
The pipeline executes three distinct search strategies through the SpotifyCrawler:
- General Search: Uses a direct
year:YYYYquery to find tracks released in a specific year. - Genre Search: Iterates through a predefined list of 18 genres (e.g., pop, rock, electronic) to find representative tracks.
- Keyword Search: Uses popularity-focused keywords (e.g., "top", "hit", "trending") to discover high-impact tracks.
Sources: service/crawler.py:34-51
Enrichment Process
Once raw track data is collected, the pipeline performs an enrichment step for the most popular tracks.
During enrichment, the pipeline fetches the primary artist's details, such as total followers, popularity, and specific genres, converting the raw API response into a structured Track object.
Sources: service/data_processor.py:58-75, service/crawler.py:118-144
Data Persistence and Export
The pipeline ensures that processed data is stored efficiently and can be shared through standardized formats.
Database Schema
The system uses a tracks table in SQLite with several key performance optimizations.
| Field | Type | Description |
|---|---|---|
track_id | str (PK) | Unique Spotify identifier (prevents duplicates). |
track_popularity | int | Popularity score (0-100). |
year | int | Indexed field for fast filtering by year. |
artist_name | str | Indexed field for artist-based queries. |
artist_genres | str | Comma-separated list of artist genres. |
Sources: README.md:38-51, app/database.py:15-39
Export and Post-Processing
The pipeline includes a dedicated export function and a post-processing script for data refinement.
- CSV Export: The
export_to_csvmethod inDataProcessorextracts data from the database and maps it to a specific set of fields for reporting. - Final Cleaning: A script (
scripts/Clean_finale.py) is used to remove test data, deduplicate, and limit results to exactly the top 200 tracks per year based on popularity.
Sources: service/data_processor.py:84-128, scripts/Clean_finale.py:10-27
Configuration and Rate Limiting
The pipeline's behavior is governed by environment variables to ensure it respects Spotify's API limits.
| Variable | Default | Purpose |
|---|---|---|
TRACKS_PER_YEAR | 1000 | Total tracks to attempt to crawl per year. |
TOP_N_TRACKS | 200 | Number of tracks to enrich and keep after filtering. |
API_RATE_LIMIT_DELAY | 0.1s | Delay between search requests to avoid 429 errors. |
ARTIST_FETCH_DELAY | 0.05s | Delay specifically for artist metadata enrichment calls. |
Sources: app/config.py:15-24, README.md:65-72
Conclusion
The Data Processing Pipeline provides a robust framework for transforming volatile API data into a structured dataset. By combining modular search strategies, batch database management, and structured data models, it ensures that the resulting music repository is both accurate and optimized for the downstream FastAPI application.