Logging and Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Logging and Monitoring system in the Spotify API Data Collector is designed to provide comprehensive visibility into the data ingestion pipeline, API performance, and system health. It utilizes a centralized logging architecture that captures multi-level diagnostic information across all service layers, including the Spotify crawler, data processor, and the FastAPI-based web interface.
The primary purpose of this system is to facilitate real-time tracking of data collection progress, identify bottlenecks in API rate limiting, and ensure the persistence of audit trails for debugging and analysis. Monitoring is further supported by dedicated health check endpoints that verify service availability and database connectivity.
Sources: app/logger.py, app/main.py:12-60, app/api.py:228-231
Logging Architecture
The project employs a structured logging approach using Python's standard logging library, enhanced with custom handlers for both file persistence and console output.
Logger Configuration
The system is initialized through setup_logger, which configures a dual-output stream:
- File Handler: Persists logs in UTF-8 encoding to the
logs/directory. It is set toDEBUGlevel to capture granular technical details. - Console Handler: Provides high-level operational feedback to the standard output (
sys.stdout) at theINFOlevel.
The logging system uses a specific format: %(asctime)s - %(levelname)s - %(name)s - %(message)s.
Sources: app/logger.py:9-40
Logging Functions and Utilities
| Function | Description | Source |
|---|---|---|
setup_logger(name, log_file) | Initializes the logger, creates the logs/ directory, and attaches file/console handlers. | app/logger.py:9-37 |
get_logger(name) | Retrieves an existing logger or initializes a new one if handlers are missing. | app/logger.py:43-50 |
logger.propagate = False | Prevents logs from being passed to the root logger to avoid duplicate entries. | app/logger.py:35, test/test_logger.py:38-42 |
System Monitoring and Health Checks
Monitoring is integrated into the FastAPI application through specialized endpoints and startup events.
API Health Endpoint
The system exposes a /health endpoint that allows external monitoring tools to verify the operational status of the service.
Sources: app/api.py:228-231
Data Ingestion Monitoring
During the collection process, the DataProcessor and SpotifyCrawler provide progress tracking:
- Batch Progress: The processor logs every 20 tracks enriched to monitor long-running jobs.
- Strategy Tracking: The crawler logs which search strategy (General, Genre, or Keyword) is currently active and the number of tracks retrieved per year.
- API Performance: The
SpotifyClientlogs rate-limiting delays and successful vs. failed API calls to the Spotify Web API.
Operational Logs
The main entry point of the application (app/main.py) uses the logger to create clear visual separators for the start and end of collection cycles.
logger.info("=" * 60)
logger.info("Démarrage du collecteur de données Spotify API")
logger.info("=" * 60)
# ... process ...
logger.info("Collecte terminée avec succès !")
Sources: app/main.py:16-18, app/main.py:53-55
Error Reporting
Errors are categorized and logged with full exception context (exc_info=True) to facilitate rapid troubleshooting:
- Configuration Errors: Logged at
ERRORlevel if Spotify credentials are missing. - API Failures: Logged at
ERRORlevel when search or artist retrieval fails. - Enrichment Failures: Logged at
WARNINGlevel if specific track metadata cannot be retrieved, allowing the process to continue.
Sources: app/main.py:24-25, service/spotify_client.py:49-51, service/data_processor.py:68-70
Conclusion
The Logging and Monitoring system provides a robust framework for managing the Spotify API Data Collector. By combining persistent file logs for deep debugging with high-level console feedback and API health endpoints, the system ensures that developers can effectively track the state of data collection and maintain high service reliability.
Sources: app/logger.py, app/api.py, service/data_processor.py