File Scanning & Deduplication

Relevant source files

The following files were used as context for generating this wiki page:

• apps/desktop/src-tauri/src/scanner.rs
• crates/core/src/metadata.rs
• crates/core/src/riff.rs
• crates/core/tests/scan_corpus.rs
• README.md
• CHANGELOG.md

File Scanning and Deduplication is the foundation of the DJ Assistant library management system. This module identifies audio files in local directories, computes unique fingerprints to prevent redundant imports, and extracts embedded metadata to populate the local SQLite database.

The system prioritizes non-destructive operations and local-first performance. It ensures that source audio files remain untouched while providing a fully searchable library based on metadata and file content hashes.

Sources: README.md:121-125, CHANGELOG.md:104-108

Scanner Architecture

The scanning process runs in a dedicated background thread to maintain UI responsiveness. It coordinates between the desktop application layer and the core Rust library to process files sequentially.

The diagram shows the sequential pipeline from folder selection to database insertion.
Sources: apps/desktop/src-tauri/src/scanner.rs:17-40, README.md:144-150

Key Components

Component	Responsibility
discover_audio_files	Recursively finds files matching supported extensions (MP3, FLAC, WAV, AIFF, M4A).
compute_file_fingerprint	Generates a BLAKE3 hash of the file content for identification.
ScanDeduplicator	Tracks fingerprints within the current batch to prevent intra-scan duplicates.
extract_metadata	Parses tags (Title, Artist, BPM, Key) using containers like ID3 and Vorbis.
cache_artwork	Writes embedded images to a content-addressed local directory.

Sources: apps/desktop/src-tauri/src/scanner.rs:26-34, crates/core/src/metadata.rs:114-140, CHANGELOG.md:113-125

Content-Based Deduplication

DJ Assistant uses BLAKE3 hashing to ensure every track in the library is unique by content, regardless of its filename or location.

Deduplication Logic

The system performs a two-tier check for every discovered file:

1. In-Memory Batch Check: The ScanDeduplicator checks if the hash has already been encountered during the current scanning session.
2. Database Check: The scanner queries the SQLite tracks table to see if a file with the same file_hash exists in the current workspace.

If the system identifies a duplicate, it increments the total_duplicates counter and emits a scan:progress event with is_duplicate: true.

Sources: apps/desktop/src-tauri/src/scanner.rs:64-92, crates/core/tests/scan_corpus.rs:180-195

Metadata Extraction & Fallbacks

Metadata extraction is designed to be "forgiving." If a file has corrupt tags, the scanner uses the filename as a fallback to ensure the file is still imported.

Extraction Pipeline

1. Primary Reader: Uses the audiotags crate for standard fields (Title, Artist, Album, Year).
2. Secondary Parsers: Direct id3 and metaflac reads capture fields not supported by audiotags, such as BPM, Camelot Key, and user ratings.
3. Container Durations: For WAV and AIFF files where audiotags often fails to report duration, custom RIFF and AIFF parsers calculate length from the file headers.
4. Filename Fallback: If no title or artist is found, the system uses the file stem as the title and "Unknown Artist" as the artist.

Sources: crates/core/src/metadata.rs:7-15, crates/core/src/metadata.rs:114-140, crates/core/src/riff.rs:27-35

Key Normalization

The system maps human-readable keys (e.g., "A minor") to the Camelot wheel system (e.g., "8A") to assist in harmonic mixing.

Database Schema for Tracks

The scanner populates the tracks table with extracted information. Every track is assigned a unique UUID and linked to a workspace_id.

Field	Type	Description
id	TEXT (UUID)	Unique primary key for the track.
file_hash	TEXT (BLAKE3)	Content-based fingerprint for deduplication.
file_path	TEXT	Absolute path to the source file.
bpm	REAL	Tempo extracted from tags (optional).
key_camelot	TEXT	Normalized Camelot key code.
artwork_hash	TEXT	BLAKE3 hash of the cached artwork for addressing.
analysis_status	TEXT	Defaults to 'pending' upon initial scan.

Sources: apps/desktop/src-tauri/src/scanner.rs:136-155, CHANGELOG.md:139-145

Conclusion

The File Scanning & Deduplication system provides a robust entry point for music libraries. By combining BLAKE3 fingerprinting with multi-container metadata extraction and container-specific duration parsers, it ensures high data integrity and library searchability. The non-destructive approach preserves original files while building a comprehensive local database for subsequent analysis and set preparation.