Specific File Analysis

Relevant source files

The following files were used as context for generating this wiki page:

• core.py
• duplicate_code_detection.py
• run_action.py
• action.yml
• examples/flext_integration_example.py
• README.md

Introduction

The Duplicate Code Detection Tool is a technical solution designed to identify similarities and duplications across source code files and directories. Its primary purpose is to enhance code quality assurance by providing a mechanism to detect significant code overlaps, which is critical for maintaining clean architectures and identifying candidates for refactoring. The tool is structured to function as both a standalone Python library and a GitHub Action, supporting seamless integration into CI/CD pipelines.

The analysis engine utilizes Natural Language Processing (NLP) techniques, specifically gensim for document similarity and nltk for tokenization, to compare files. It supports multiple programming languages including Python, C, C++, Java, and JavaScript.

Sources: README.md:1-25, duplicate_code_detection.py:12-25

Architecture and Core Components

The system is organized into a core processing layer, a high-level API wrapper, and integration interfaces for CLI and GitHub Actions.

Core Logic Engine

The analysis is driven by the run function in duplicate_code_detection.py, which handles file discovery, content parsing (including optional comment removal), and similarity computation.

The diagram above illustrates the data flow from raw file input to the generation of the similarity matrix.
Sources: duplicate_code_detection.py:228-335, core.py:121-150

High-Level Library Interface

The core.py module introduces the DuplicateCodeDetector class, which wraps the low-level run function into a developer-friendly API. It utilizes several data classes to structure the results.

Component	Type	Description
DuplicateCodeDetector	Class	Main entry point for library usage; manages configuration like thresholds and file extensions.
FileSimilarity	DataClass	Represents a pair-wise similarity result between two specific files.
SimilarityReport	DataClass	Aggregates all similarity findings for a single source file.
DetectionResult	DataClass	The final output object containing success status and all generated reports.

Sources: core.py:25-95, core.py:98-118

Detailed Logic and Implementation

Source Code Pre-processing

When the --only-code flag is set, the tool uses Python's ast and astor libraries to strip comments and docstrings from .py files, ensuring that the similarity analysis focuses strictly on functional logic.

def remove_comments_and_docstrings(source_code: str) -> str:
    parsed = ast.parse(source_code)
    for node in ast.walk(parsed):
        if not isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef, ast.Module)):
            continue
        # ... logic to remove string expressions at the start of bodies
    return astor.to_source(parsed)

Sources: duplicate_code_detection.py:65-95

Similarity Computation

The tool calculates similarity using the following sequence:

1. Tokenization: Converting source code into lowercase word tokens.
2. Vectorization: Converting tokens into a Bag-of-Words (BoW) representation.
3. TF-IDF Transformation: Weighting tokens based on their frequency across the corpus.
4. Cosine Similarity: Comparing document vectors via gensim.similarities.Similarity.

Sources: duplicate_code_detection.py:284-301

Integration and Interfaces

GitHub Action Integration

The tool provides a Docker-based GitHub Action. The run_action.py script acts as a bridge, executing the detection and posting the results back to GitHub Pull Requests as Markdown comments.

Configuration Parameters

The following table summarizes the primary configuration options available across the CLI and Library interfaces:

Parameter	Default	Library Key	CLI Flag
Fail Threshold	100	fail_threshold	-t / --fail-threshold
Ignore Threshold	15	ignore_threshold	--ignore-threshold
Only Code	False	only_code	--only-code
Show LOC	True	show_loc	--show-loc
File Extensions	["py", "js", ...]	file_extensions	--file-extensions

Sources: core.py:108-118, duplicate_code_detection.py:130-180

Conclusion

The Duplicate Code Detection Tool provides a robust architecture for identifying code redundancy. By combining high-level abstraction in core.py with the powerful analysis capabilities of gensim in duplicate_code_detection.py, it serves as an effective quality gate for both individual developers and automated CI systems. The inclusion of specialized pre-processing and multiple output formats (JSON, CSV, Markdown) makes it highly adaptable to various workflow requirements.

Sources: README.md:270-285, examples/flext_integration_example.py:175-190