Similarity Calculation Logic

Relevant source files

The following files were used as context for generating this wiki page:

• duplicate_code_detection.py
• core.py
• run_action.py
• README.md
• action.yml

The Similarity Calculation Logic is the core engine of the Duplicate Code Detection Tool. It identifies and quantifies the degree of similarity between source code files within a repository. This logic is used to maintain code quality by identifying redundant code blocks that should be refactored into reusable components.

The system utilizes Natural Language Processing (NLP) techniques, specifically Term Frequency-Inverse Document Frequency (TF-IDF) and Cosine Similarity, to compare document vectors. It processes files by tokenizing content, converting it into a bag-of-words representation, and then calculating similarity scores against a generated corpus index.
Sources: duplicate_code_detection.py:1-10, README.md:10-25

Core Processing Pipeline

The similarity detection process follows a linear pipeline: file discovery, preprocessing (optional cleaning), vectorization, and similarity scoring.

The diagram above shows how raw source code is transformed into a searchable similarity index.
Sources: duplicate_code_detection.py:215-240

Preprocessing and Tokenization

Before calculation, files can be preprocessed to focus solely on functional logic. If the only_code flag is enabled, the tool utilizes Python's ast module to strip docstrings and comments from .py files.
Sources: duplicate_code_detection.py:65-95, core.py:115-125

Step	Method	Description
Cleaning	remove_comments_and_docstrings	Uses ast.parse and ast.walk to remove non-functional string expressions.
Tokenization	word_tokenize	Converts raw strings into lowercase word tokens.
Vectorization	doc2bow	Converts tokens into a Bag-of-Words (BoW) format (token_id, token_count).

Sources: duplicate_code_detection.py:65-95, 215-225, core.py:15-20

Vector Space Modeling

The tool employs the gensim library to create a vector space model. This involves three primary components:

1. Dictionary: A mapping of every unique word found across all files to a unique integer ID.
2. Corpus: A collection of BoW vectors representing every file in the analysis set.
3. TF-IDF Model: A model that weighs tokens based on their frequency within a document relative to their frequency across the entire corpus, ensuring common language keywords have less impact than unique logic.

Sources: duplicate_code_detection.py:220-230, requirements.txt:1

Similarity Calculation Logic

The calculation is performed using gensim.similarities.Similarity. For every file (the "query"), the tool calculates its similarity against the rest of the corpus.

The sequence diagram illustrates how the run() function iterates through files to perform cross-comparisons against the indexed corpus.
Sources: duplicate_code_detection.py:215-260

Configuration Parameters

The logic is controlled by several thresholds that determine which results are reported and whether the process is considered a failure.

Parameter	Type	Source	Description
fail_threshold	int	action.yml:32	Maximum allowed similarity (default 100) before exiting with ReturnCode.THRESHOLD_EXCEEDED.
ignore_threshold	int	action.yml:28	Minimum similarity percentage (default 10) to be included in reports.
only_code	bool	action.yml:41	If true, strips Python comments and docstrings before analysis.
file_extensions	list	duplicate_code_detection.py:20	Filters files considered for the corpus (e.g., .py, .java, .cpp).

Sources: duplicate_code_detection.py:20, 115-150, action.yml:20-45

Result Processing and Output

Once similarities are calculated, the results are aggregated into structured objects for reporting. In the core.py module, these are represented by the DetectionResult and SimilarityReport classes.

# From core.py:25-30
@dataclass
class FileSimilarity:
    """Represents similarity between two files."""
    source_file: str
    target_file: str
    similarity_percentage: float
    source_loc: int | None = None
    target_loc: int | None = None

Sources: core.py:25-45

Output Transformation

The raw similarity scores (floats between 0.0 and 1.0) are converted to percentages. Results are filtered based on the ignore_threshold and sorted to present the most significant duplications first.
Sources: duplicate_code_detection.py:270-290, run_action.py:100-110

Conclusion

The Similarity Calculation Logic provides a robust mechanism for detecting code reuse. By combining AST-based preprocessing with TF-IDF vectorization, the tool moves beyond simple text matching to identify structural similarities in code across multiple programming languages.
Sources: duplicate_code_detection.py:1-5, 215-230, core.py:150-160