Audio Analysis Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Analysis Engine is the core component of the MoodTune backend responsible for extracting digital signal processing (DSP) features from audio files to determine musical mood. It utilizes the librosa library to analyze raw audio data from three primary sources: direct user uploads, Jamendo catalog streams, and Spotify preview metadata. The engine transforms acoustic characteristics—such as tempo, spectral centroid, and RMS energy—into high-level psychological attributes like valence, energy, and danceability.
The engine operates through two main methodologies: direct audio analysis via AudioAnalysisService for local/streamed files and statistical playlist analysis via analysis_service.py for Spotify-sourced metadata. This system enables the application to provide "mood-forward" insights, including human-readable labels for texture, brightness, and energy levels.
Sources: app/services/audio_analysis_service.py:15-20, app/api/v1/catalog.py:27-35, app/services/analysis_service.py:10-15
System Architecture and Data Flow
The architecture follows a pipeline where raw audio bytes or URLs are ingested, processed through temporary storage, and analyzed to produce structured JSON responses.
The diagram illustrates the separate pathways for raw audio processing versus metadata-based classification.
Sources: app/services/audio_analysis_service.py:32-45, app/api/v1/catalog.py:36-40, app/api/v1/audio_upload.py:44-55
Audio Feature Extraction Pipeline
The engine extracts a variety of low-level features to build a comprehensive acoustic profile. For direct audio files, librosa is configured to load the first 30 seconds for standard analysis or up to 300 seconds for detailed uploads.
Sources: app/services/audio_analysis_service.py:53-55, app/services/audio_analysis_service.py:100-105
Core Acoustic Features
| Feature | Implementation | Description |
|---|---|---|
| Tempo | librosa.beat.beat_track | Estimated Beats Per Minute (BPM). |
| Energy | librosa.feature.rms | Root Mean Square energy representing loudness. |
| Brightness | librosa.feature.spectral_centroid | The "center of mass" of the spectrum; higher values indicate brighter sounds. |
| Chroma | librosa.feature.chroma_stft | Represents the 12 different pitch classes; used for harmonic analysis. |
| Zero Crossing Rate | librosa.feature.zero_crossing_rate | Measure of percussiveness and signal noisiness. |
| MFCC | librosa.feature.mfcc | Mel-frequency cepstral coefficients representing timbre. |
Sources: app/services/audio_analysis_service.py:65-90, app/services/audio_analysis_service.py:137-170
Mood Classification Logic
Mood is determined by mapping acoustic features onto a two-dimensional Valence-Energy model.
Valence and Energy Estimation
Valence (positivity/happiness) and Danceability are derived using heuristic formulas based on extracted features:
- Valence Estimation: Calculated as a weighted combination of normalized brightness (60%) and normalized tempo (40%). Brighter and faster tracks result in higher valence.
- Danceability Estimation: Calculated based on proximity to an "optimal" dance tempo (110 BPM) and raw energy levels.
Sources: app/services/audio_analysis_service.py:120-135, IMPROVEMENTS.md:46-50
Mood Quadrants
The engine classifies tracks into five primary moods based on the following thresholds:
| Mood | Valence Range | Energy Range | Center Point |
|---|---|---|---|
| Happy | > 0.6 | > 0.6 | V: 0.7, E: 0.7 |
| Sad | < 0.4 | < 0.4 | V: 0.3, E: 0.3 |
| Angry | < 0.4 | > 0.6 | V: 0.3, E: 0.9 |
| Calm | Any | < 0.4 | V: 0.5, E: 0.2 |
| Energetic | Default Case | > 0.6 | V: 0.6, E: 0.9 |
Sources: app/services/audio_analysis_service.py:22-30, app/services/audio_analysis_service.py:222-245
Statistical Playlist Analysis
For Spotify playlists, the analysis_service.py performs aggregate calculations across multiple tracks to determine the "Primary Mood" of the collection.
The confidence score is normalized to a 30-100% scale, where higher distance from the neutral center (0.5, 0.5) indicates higher confidence in the classification.
Sources: app/services/analysis_service.py:75-90, app/services/analysis_service.py:158-185
Human-Readable Descriptive Labels
The service includes logic to translate raw floating-point data into UI-friendly tags and labels to improve user experience.
- Energy Label: Categorized as High (> 0.15 RMS), Medium (> 0.05), or Low.
- Brightness Label: Categorized as Bright (> 3000 Hz spectral centroid), Warm (> 1500 Hz), or Dark.
- Texture Label: Categorized as Smooth (> 0.6 harmonic ratio), Textured (> 0.3), or Rough.
- Mood Descriptors: Generates short tags like "driving," "melancholic," or "fast-paced" based on combined feature sets.
Sources: app/services/audio_analysis_service.py:268-305, app/schemas/audio_analysis.py:30-33
Technical Implementation Details
API Integration
The engine is exposed via multiple endpoints tailored for different use cases:
- Catalog Analysis:
POST /api/v1/catalog/analyzedownloads remote MP3s and runs the full Librosa pipeline. - Audio Upload:
POST /api/v1/audio-upload/analyzehandles in-memory file processing for guest users. - Spotify Analysis:
POST /api/v1/analysis/analyze/{playlist_id}uses pre-calculated Spotify metadata.
Sources: app/api/v1/catalog.py:21-30, app/api/v1/audio_upload.py:16-25
Error Handling and Fallbacks
To prevent system crashes on corrupted audio, the service implements default feature sets. If analysis fails, it returns a neutral profile (Tempo: 120, Energy: 0.5, Valence: 0.5).
Sources: app/services/audio_analysis_service.py:212-220, IMPROVEMENTS.md:76-80
Performance Considerations
The service utilizes a temporary file system managed by tempfile to handle librosa loading requirements. Files are immediately unlinked (deleted) after the feature extraction completes to minimize disk usage.
Sources: app/services/audio_analysis_service.py:202-210, app/services/audio_analysis_service.py:317-325
Future Improvements
Identified areas for engine enhancement include:
- Adaptive Normalization: Replacing hardcoded RMS limits (0.5) with track-specific min/max values.
- Logarithmic Scaling: Using log scales for RMS to better match human loudness perception.
- Key Detection: Adding musical key detection (Major keys generally correlate with higher valence).
Sources: IMPROVEMENTS.md:5-25, IMPROVEMENTS.md:55-60