Deduplication Engine

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/dedup_engine.py
• scripts/tests/test_dedup_engine.py
• scripts/generate_knowledge_base.py
• knowledge/papers/a-data-driven-methodology-for-quality-aware-code-fixing.json
• knowledge/techniques/quality-aware-code-fixing.json

The Deduplication Engine is a core utility within the CodeScholar project designed to prevent the insertion of duplicate research papers when aggregating data from multiple external sources. It ensures the integrity of the unified library by utilizing a multi-tiered matching strategy that ranges from high-confidence identifier checks to fuzzy string comparisons of titles.

This system is particularly critical during the Knowledge Base Generation process, where metadata is fetched and transformed into a structured 3-layer architecture. By filtering out redundant entries, the engine maintains a clean dataset for the downstream AI extraction passes.

Sources: scripts/dedup_engine.py:1-12, scripts/generate_knowledge_base.py:27-35

Architecture and Components

The engine is implemented as a centralized class that manages multiple lookup indexes for fast retrieval and comparison. It operates primarily on "paper" objects, which are dictionaries containing metadata and unique identifiers.

Matching Strategies

The engine follows a specific hierarchy of confidence when evaluating whether a new paper already exists in the library:

1. DOI Exact Match: The highest confidence level, using normalized Digital Object Identifiers.
2. arXiv ID Exact Match: Matching based on unique preprint identifiers.
3. Semantic Scholar ID Match: Utilizing the s2_id from the Semantic Scholar API.
4. Title Fuzzy Match: A fallback mechanism using Levenshtein distance to detect identical or nearly identical titles.

Sources: scripts/dedup_engine.py:7-12, scripts/dedup_engine.py:126-170

Data Flow for Duplicate Detection

The following diagram illustrates how a paper is processed by the engine to determine its uniqueness.

The engine checks identifiers in order of decreasing specificity to ensure the most reliable match is found first.

Sources: scripts/dedup_engine.py:130-186

Implementation Details

Core Classes and Data Structures

Class/Struct	Purpose	Key Attributes
DeduplicationEngine	Main coordinator for indexing and matching.	doi_index, arxiv_index, title_index
DeduplicationResult	Dataclass representing the outcome of a check.	is_duplicate, match_type, confidence

The DeduplicationEngine maintains internal dictionaries (e.g., Dict[str, str]) where keys are normalized identifiers or titles, and values are the corresponding internal paper_id.

Sources: scripts/dedup_engine.py:17-25, scripts/dedup_engine.py:86-93

Normalization Logic

Before any comparison occurs, data is passed through normalization functions to remove noise such as URL prefixes, casing, and punctuation.

• DOI Normalization: Converts to lowercase and strips prefixes like https://doi.org/.
• Title Normalization: Converts to lowercase, removes all non-alphanumeric characters, and collapses whitespace.

Sources: scripts/dedup_engine.py:28-50, scripts/tests/test_dedup_engine.py:20-50

Similarity Scoring

For title comparisons that are not exact, the engine uses the Levenshtein distance algorithm to calculate a similarity score between 0.0 and 1.0.

# Sources: [scripts/dedup_engine.py:72-83]
def title_similarity(title1: str, title2: str) -> float:
    norm1 = normalize_title(title1)
    norm2 = normalize_title(title2)
    # ...
    distance = levenshtein_distance(norm1, norm2)
    return 1.0 - (distance / max_len)

The default title_threshold for considering a paper a duplicate is 0.9 (90% similarity).

Sources: scripts/dedup_engine.py:121-125, scripts/dedup_engine.py:170-184

Batch Processing

The engine supports deduplicate_batch, which handles two types of collisions:

1. Library Collisions: New papers that match papers already stored in the system.
2. Intra-batch Collisions: Duplicate papers appearing multiple times within the same incoming dataset.

This ensures that even if a paper is not in the library yet, it won't be added twice if it appears multiple times in the same source.

Sources: scripts/dedup_engine.py:192-243, scripts/tests/test_dedup_engine.py:210-225

Integration with Knowledge Base Generation

The generate_knowledge_base.py script leverages the results of deduplication (specifically the identification of unique papers and techniques) to build its 3-layer architecture.

• Pass 1: Identifies unique papers for GPT-4 extraction.
• Pass 2: Clusters these unique papers by technique names to generate synthesis cards.
• Technique Deduplication: Technique names are stripped and normalized to group findings across different research papers.

Sources: scripts/generate_knowledge_base.py:530-550, knowledge/techniques/quality-aware-code-fixing.json:5-15

Summary

The Deduplication Engine provides a robust mechanism for maintaining data integrity in CodeScholar. By combining strict identifier matching (DOI, arXiv) with flexible fuzzy title matching, it effectively handles the inconsistencies common in academic metadata aggregation. This ensures that the downstream Knowledge Base remains concise, accurate, and free of redundant information.