Utility and Processors Tests

Relevant source files

The following files were used as context for generating this wiki page:

• test/test_config.py
• test/test_data_processor.py
• test/test_logger.py
• test/test_db.py
• test/test_main.py
• test/test_crawler_extended.py
• app/config.py

The Utility and Processors Tests suite ensures the reliability of the core logic responsible for data transformation, environment configuration, system logging, and database connectivity. This module covers the validation of the application's infrastructure components, ensuring that data retrieved from the Spotify API is correctly processed, stored, and exported.

The scope of these tests includes the orchestration of the collection process through the DataProcessor, the integrity of the Config management system, and the robustness of the Database utility wrapper. By simulating various operational states—including successful executions, API errors, and configuration failures—the test suite guarantees that the system behaves predictably under different conditions.

Sources: test/test_main.py:11-13, test/test_data_processor.py:43-45, test/test_config.py:10-12

Data Processing and Orchestration

The DataProcessor serves as the central orchestrator that manages the flow of information between the SpotifyCrawler and the persistence layers (SQLite and CSV). The tests verify that the processor correctly handles batching, data enrichment, and conditional database initialization.

Data Flow Logic

The processor is tested for its ability to iterate through target years, invoke the crawler for raw track data, filter those results by popularity, and enrich them with artist metadata. The tests ensure that errors during the enrichment phase (e.g., individual artist fetch failures) do not halt the entire collection process; instead, failed tracks are logged and skipped.

The diagram above illustrates the sequence of operations managed by the DataProcessor during a standard collection cycle.
Sources: test/test_data_processor.py:72-100, test/test_data_processor.py:121-140

CSV Export Capabilities

Tests validate that the DataProcessor can export data from two sources: an in-memory list of tracks or directly from the SQLite database. This ensures flexibility for both the main collection script and standalone export utilities.

Feature	Description	Source File
In-Memory Export	Exports a provided list of Track objects to a CSV file.	test/test_data_processor.py:186-210
Database Export	Queries all tracks from the DB and writes them to a CSV file.	test/test_data_processor.py:212-250
Empty Handling	Ensures the system does not crash when exporting an empty dataset.	test/test_data_processor.py:175-184

Configuration Management

The Config class is responsible for loading environment variables from .env and providing default values for operational parameters. Tests ensure that the application fails fast if critical credentials (SPOTIPY_CLIENT_ID, SPOTIPY_CLIENT_SECRET) are missing.

Validation Rules

The Config.validate() method is tested against various failure modes:

• Missing Client ID or Secret (raises ValueError).
• Empty string credentials (raises ValueError).
• Successful validation when both are present.

Sources: test/test_config.py:28-60, app/config.py:23-31

Parameter Summary

The following configuration attributes are verified for existence and correct type assignment:

Parameter	Type	Default Value	Description
TRACKS_PER_YEAR	int	1000	Max tracks searched per year
TOP_N_TRACKS	int	200	Tracks kept after popularity filter
YEARS	list	[2020...2024]	Target years for collection
DATABASE_URL	str	sqlite:///spotify_data.db	Connection string
API_RATE_LIMIT_DELAY	float	0.1	Delay between API search calls

Sources: app/config.py:14-21, test/test_config.py:14-23

Database Utility and Persistence

The Database utility acts as a wrapper around sqlite3, providing a thread-safe context manager for connections and helper methods for common operations. Tests focus on connection lifecycle management and transaction integrity.

Connection Management

Tests confirm that the Database class:

1. Automatically creates the necessary parent directories for the database file if they do not exist.
2. Properly rolls back transactions if an exception occurs within a with get_connection() block.
3. Correctly converts row results into dictionary-like objects for easier access.

This diagram demonstrates the transaction safety verified in the database utility tests.
Sources: test/test_db.py:44-64, test/test_db.py:136-152

Logging Infrastructure

The setup_logger utility ensures consistent log formatting and output across the application. Tests verify that the system creates two handlers: one for the console (stdout) and one for a rotating log file.

Logger Verification

The tests ensure the following properties of the logging system:

• Singleton-like behavior: Multiple calls to get_logger with the same name return the same object to prevent handler duplication.
• Propagation: Propagation to the root logger is disabled (propagate = False) to avoid duplicate logs in nested environments.
• Levels: The default level is set to DEBUG for comprehensive troubleshooting.

Sources: test/test_logger.py:11-20, test/test_logger.py:53-62

Main Execution Workflow

The orchestration of all these utilities is tested via the main.py entry point. Mocking is extensively used to ensure that the main() function correctly sequences the initialization of the Spotify client, the crawler, and the processor.

Process Sequence

1. Validate Config: Ensure credentials exist.
2. Initialize Services: Setup Client, Crawler, and Processor.
3. Collect: Execute collect_data_for_years().
4. Export: Trigger export_to_csv().

Sources: test/test_main.py:16-43, app/main.py:21-50

Conclusion

The Utility and Processors Tests provide a safety net for the application's foundational services. By validating configuration, data processing pipelines, and database reliability, the suite ensures that the Spotify API data collection remains consistent and error-resilient throughout the 2020-2024 data lifecycle.