Troubleshooting & Maintenance
Relevant source files
The following files were used as context for generating this wiki page:
The Troubleshooting & Maintenance module provides essential tools and procedures for ensuring the health, reliability, and data integrity of the Spotify API Data Collector system. It encompasses system health monitoring, automated diagnostic tests, configuration validation, and data cleaning scripts designed to handle common failure points such as API rate limits and database corruption.
This system is built with a focus on observability through comprehensive logging and health check endpoints. Maintenance tasks include regular database indexing, CSV exports for external analysis, and the execution of a test suite to verify the integrity of the crawler and API components.
Sources: README.md:1-25, app/main.py:10-15, app/api.py:192-195
System Health & Monitoring
The system provides multiple layers of monitoring to ensure the collection process and the FastAPI exposure layer are functioning correctly.
API Health Endpoint
The web service includes a dedicated /health endpoint that returns the operational status of the service. This is the primary mechanism for automated monitoring tools to verify the availability of the Spotify API Data Collector.
The health check ensures the service is reachable and responding to requests.
Sources: app/api.py:192-195, test/test_api.py:214-222
Logging Infrastructure
The system uses a centralized logging mechanism, initialized via setup_logger, to track operations across the crawler, client, and processor. Logs are stored in a logs/ directory.
| Component | Log Level | Purpose |
|---|---|---|
spotify_collector | INFO/DEBUG | Main execution flow and setup steps |
spotify_collector.client | DEBUG | API request details and rate limit tracking |
DataProcessor | DEBUG | Enrichment progress (every 20 tracks) |
Sources: app/main.py:9-10, service/spotify_client.py:12, test/test_data_processor.py:114-125
Data Integrity & Maintenance
Maintenance of the local SQLite storage involves validating data types, ensuring idempotence during collection, and periodic cleaning of test data.
Database Stability
The system ensures stability through automated initialization and transaction management. The Database class in app/db.py handles connections and provides rollbacks on errors to prevent data corruption.
This flow prevents partial data writes during failures.
Sources: app/db.py:46-64, app/database.py:10-15
Data Cleaning and Optimization
A specific script, scripts/Clean_finale.py, is used to maintain data quality by removing test entries and enforcing popularity-based limits per year.
| Task | Operation | Target |
|---|---|---|
| Test Removal | Filters out "Test Artist" or "Test" track names | artist_name, track_name |
| Pruning | Keeps only the Top 200 tracks per year by popularity | track_popularity |
| Export | Saves refined data to spotify_data_final.csv | Output File |
Sources: scripts/Clean_finale.py:12-25, README.md:65-67
Troubleshooting Common Issues
Configuration Failures
The most common startup failure is missing Spotify API credentials. The Config.validate() method is used at the entry point of both the main crawler and the API to catch these errors early.
# Validation logic in app/config.py
@classmethod
def validate(cls):
if not cls.SPOTIPY_CLIENT_ID or not cls.SPOTIPY_CLIENT_SECRET:
raise ValueError(
"SPOTIPY_CLIENT_ID et SPOTIPY_CLIENT_SECRET doivent être définis dans le fichier .env"
)
Sources: app/config.py:22-29, app/main.py:20-25
Rate Limiting and API Errors
The system implements manual delays to avoid 429 Too Many Requests errors from the Spotify API.
- API Rate Limit Delay:
API_RATE_LIMIT_DELAY(default 0.1s) is applied after search queries and track fetches. - Artist Fetch Delay:
ARTIST_FETCH_DELAY(default 0.05s) is applied after retrieving artist metadata.
If the API returns an error, the SpotifyClient logs the exception with a full stack trace (exc_info=True) to facilitate debugging.
Sources: service/spotify_client.py:38-40, service/spotify_client.py:53-55, app/config.py:18-19
Diagnostic Testing
Developers should run the test suite to diagnose issues with specific modules:
# Run all tests with coverage to identify untested failure paths
python3.11 -m pytest test/ --cov=app --cov=service --cov-report=term-missing
Sources: README.md:120-125, test/test_api.py:1-10
Summary
Troubleshooting and maintenance of the Spotify API Data Collector rely on a combination of configuration validation, defensive database operations, and post-collection data refining. By utilizing the health endpoint and analyzing logs in the logs/ directory, administrators can identify and resolve API connectivity or rate-limiting issues efficiently. Regular execution of the Clean_finale.py script ensures that the production database remains free of test data and focuses on the highest-quality musical metadata.
Sources: README.md:140-150, app/api.py:200-210