Audio Analysis Improvements
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Analysis module in the MoodTune backend is responsible for extracting musical features and determining the mood of audio tracks through librosa. While the current implementation provides a foundation for mood classification based on valence, energy, and tempo, several areas for refinement have been identified to enhance accuracy, performance, and reliability. These improvements target the core analysis pipeline, including normalization strategies, feature extraction depth, and error handling for diverse audio sources.
The system currently supports analysis via three main entry points: user-uploaded files, Jamendo catalog streams, and Spotify audio features. Improvements are focused on the direct audio processing pipeline used by AudioAnalysisService, which handles raw audio data to produce human-readable mood descriptors and technical metrics.
Current Analysis Pipeline and Identified Issues
The existing pipeline utilizes a fixed set of weights and hardcoded normalization values to estimate mood. Specifically, energy normalization relies on a hardcoded range (0 to 0.5), which can lead to data loss or inconsistency across varying audio sources.
Audio Analysis Flow
The following diagram illustrates the current flow of audio data through the analysis service.
Identified Bottlenecks
| Area | Current Limitation | Proposed Improvement |
|---|---|---|
| Energy Normalization | Uses hardcoded (0, 0.5) range for RMS values. | Adaptive normalization using rms.min() and rms.max(). |
| Valence Estimation | Fixed weights (60% brightness, 40% tempo). | Incorporate chroma features and key detection (major vs minor). |
| Performance | Downloads full previews and lacks caching. | Implement streaming, chunked processing, and result caching. |
| Error Handling | Returns default features on any error. | Granular error types and partial result returns. |
Sources: IMPROVEMENTS.md:5-80, app/services/audio_analysis_service.py:168-185
Feature Extraction Enhancements
The system extracts several key features to determine mood, but there is potential to utilize more complex indicators such as Zero Crossing Rate and Spectral Rolloff to improve texture and brightness labeling.
Enhanced Feature Set
Currently, the _extract_upload_features method captures a wider array of metrics than the standard analyze_audio_file method. The goal is to standardize these across all analysis paths.
# app/services/audio_analysis_service.py:189-210
def _extract_upload_features(self, y: np.ndarray, sr: int, duration_seconds: float) -> Dict:
tempo, beat_frames = librosa.beat.beat_track(y=y, sr=sr)
onset_env = librosa.onset.onset_strength(y=y, sr=sr)
beat_strength = float(np.mean(onset_env) / (np.max(onset_env) + 1e-9))
spectral_centroid = librosa.feature.spectral_centroid(y=y, sr=sr)
spectral_rolloff = librosa.feature.spectral_rolloff(y=y, sr=sr)
y_harmonic, y_percussive = librosa.effects.hpss(y)
harm_rms = float(np.mean(librosa.feature.rms(y=y_harmonic)))
perc_rms = float(np.mean(librosa.feature.rms(y=y_percussive)))
harmonic_ratio = harm_rms / (harm_rms + perc_rms + 1e-9)
Proposed Logic for Mood Descriptors
Refining the _mood_descriptors function involves moving beyond static tags to dynamic labels based on threshold-based feature analysis.
| Feature | Condition | Resulting Tag |
|---|---|---|
| Tempo | > 140 BPM | "fast-paced" |
| Harmonic Ratio | > 0.7 | "smooth" |
| Danceability | > 0.7 | "danceable" |
| Acousticness | > 0.5 | "acoustic" |
Sources: app/services/audio_analysis_service.py:355-373, app/services/analysis_service.py:186-198
Architecture for Improved Reliability
To prevent the "default feature masking" issue, the architecture should shift toward a pipeline that validates audio integrity before processing.
Audio Validation & Processing Sequence
Summary of Planned Data Model Improvements
The AudioAnalysisResponse schema represents the target for analysis output. Improvements will focus on populating the mood_scores and descriptors fields with more granular data derived from the new feature extraction methods.
Key Schema Fields
- mood_scores: A dictionary containing normalized valence, energy, tempo, and brightness.
- audio_features: A detailed object including
spectral_rolloff,zero_crossing_rate, andmfcc_mean. - reasoning: A human-readable string explaining the mood classification logic.
Sources: app/schemas/audio_analysis.py:36-70, app/services/audio_analysis_service.py:317-325
The proposed "Audio Analysis Improvements" aim to transition the service from a heuristic-based model to a more robust, adaptive system. By implementing adaptive normalization and expanding the utilized feature set, the backend will provide more accurate and consistent mood classifications across diverse musical genres and recording qualities.