Getting Started

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• setup.py
• duplicate_code_detection.py
• core.py
• action.yml
• examples/flext_integration_example.py

The Duplicate Code Detection Tool is a technical utility designed to identify code similarities and duplications within software repositories. It supports Python 3.8+ and is optimized for integration into CI/CD pipelines, pre-commit hooks, and manual code quality audits. The tool leverages NLP techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) to calculate similarity scores between files.

Sources: README.md:1-20, duplicate_code_detection.py:270-285

Installation and Setup

Users can install the tool either via PyPI or directly from the GitHub repository, which is the recommended method for Flext projects.

# From PyPI
pip install duplicate-code-detection-tool

# From GitHub (Recommended for Flext projects)
pip install git+https://github.com/flext-sh/duplicate-code-detection-tool.git

Sources: README.md:22-34, setup.py:16-47

Dependencies

The tool requires the following core dependencies for text processing and code analysis:

• gensim>=4.0.0: Used for document similarity modeling.
• nltk>=3.8: Used for tokenization.
• astor>=0.8.1: Used for AST manipulation when stripping comments.
• numpy>=1.21.0: Supporting numerical operations.

Sources: pyproject.toml:44-48, requirements.txt:1-3

Core Architecture and Data Flow

The tool operates by tokenizing source code, converting it into a bag-of-words representation, and then applying a TF-IDF model to determine similarity percentages.

The diagram above shows the data processing pipeline from raw source code to similarity analysis.
Sources: duplicate_code_detection.py:246-285, core.py:186-215

Usage Modes

1. Library Integration

The DuplicateCodeDetector class in core.py provides the high-level API for programmatic use.

Parameter	Type	Default	Description
fail_threshold	int	100	Similarity % above which the result is considered a failure.
ignore_threshold	int	15	Minimum similarity % to report.
file_extensions	list	[py, js, ts, etc.]	File types to include in analysis.
only_code	bool	False	If true, strips Python comments and docstrings via AST.
show_loc	bool	True	Includes line count (#LoC) in the output.

Sources: core.py:136-160, README.md:154-162

2. Command Line Interface (CLI)

The CLI allows users to analyze directories or specific files directly from the terminal.

# Analyze directories with a specific ignore threshold
duplicate-code-detection -d src/ --ignore-threshold 15

# Output results to JSON for CI/CD integration
duplicate-code-detection -d src/ -j --fail-threshold 80

Sources: README.md:73-86, duplicate_code_detection.py:145-200

3. GitHub Actions

The tool can be run as a Docker-based GitHub Action, providing automated reports in Pull Request comments.

The sequence diagram illustrates how the tool integrates into a GitHub Action environment to report duplications.
Sources: action.yml:1-55, entrypoint.sh:5-24, run_action.py:130-180

Result Processing Logic

Analysis results are encapsulated in the DetectionResult class, which contains a list of SimilarityReport objects.

# from core.py:84-118
@dataclass
class DetectionResult:
    return_code: ReturnCode
    reports: list[SimilarityReport] = field(default_factory=list)
    raw_data: dict[str, Any] = field(default_factory=dict)

    def is_success(self) -> bool:
        return self.return_code == ReturnCode.SUCCESS

    def get_critical_duplications(self, threshold: float = 50.0):
        # Filters similarities above the provided threshold
        ...

Sources: core.py:84-125, duplicate_code_detection.py:43-47

Configuration and Exit Codes

The tool uses standard exit codes to signal the result of the analysis:

• ReturnCode.SUCCESS (0): No duplications found above the fail_threshold.
• ReturnCode.BAD_INPUT (1): Invalid directory paths or insufficient file counts (minimum 2 files required).
• ReturnCode.THRESHOLD_EXCEEDED (2): Similarity detected above the user-defined fail_threshold.

Sources: duplicate_code_detection.py:43-47, duplicate_code_detection.py:220-245

Summary

The Duplicate Code Detection Tool provides a robust framework for identifying code duplication through CLI, library, and GitHub Action interfaces. By utilizing TF-IDF and NLTK tokenization, it offers precise similarity metrics that can be configured to ignore boilerplate code or comments, making it an essential component for maintaining code quality in large-scale Python and multi-language projects.

Sources: README.md:5-15, core.py:10-30