Home Page

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• duplicate_code_detection.py
• core.py
• action.yml
• run_action.py
• examples/flext_integration_example.py
• pyproject.toml

The Duplicate Code Detection Tool is a Python-based utility designed to identify similarities and duplications across source code files and directories. It serves as both a standalone library and a command-line interface (CLI) tool, facilitating code quality assurance and integration into CI/CD pipelines. The tool supports multiple programming languages and provides configurable thresholds for reporting and failure conditions.

Sources: README.md:1-20, pyproject.toml:10-25

Core Architecture and Logic

The system is built upon a core analysis engine that utilizes natural language processing (NLP) techniques to compare document similarity. It leverages the gensim library for TF-IDF (Term Frequency-Inverse Document Frequency) modeling and nltk for tokenization.

Logic Flow

The detection process involves gathering source files, preprocessing them (optionally removing comments), tokenizing the content, and calculating similarity scores using a vector space model.

The diagram shows the sequence of operations from file discovery to similarity calculation.
Sources: duplicate_code_detection.py:215-280, core.py:155-180

Key Data Structures

The tool organizes results through a hierarchy of dataclasses defined in the core module.

Class	Description	Key Attributes
FileSimilarity	Represents a pair-wise similarity between two files.	source_file, target_file, similarity_percentage
SimilarityReport	Aggregates all similarities for a single analyzed file.	file_path, similarities, loc_count
DetectionResult	The high-level object containing the full analysis state.	return_code, reports, raw_data

Sources: core.py:24-85

Analysis Components

DuplicateCodeDetector Class

The DuplicateCodeDetector is the primary high-level interface for library users. It encapsulates configuration settings such as thresholds and file extensions.

• Initialization Parameters:
  • fail_threshold: Maximum similarity percentage allowed before returning an error code.
  • ignore_threshold: Minimum similarity percentage required to include a match in the report.
  • only_code: Boolean to toggle the removal of Python comments and docstrings.
  • show_loc: Boolean to include Lines of Code (LoC) metrics in the output.

Sources: core.py:101-140

Code Preprocessing

For Python files, the tool can perform AST-based (Abstract Syntax Tree) transformation to strip comments and docstrings, ensuring that only functional code logic is compared.

def remove_comments_and_docstrings(source_code: str) -> str:
    parsed = ast.parse(source_code)
    for node in ast.walk(parsed):
        if not isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef, ast.Module)):
            continue
        # ... logic to strip docstrings ...
    source_code_clean = astor.to_source(parsed)
    return source_code_clean

Sources: duplicate_code_detection.py:73-104

Integration Methods

GitHub Actions Integration

The tool is packaged as a Docker-based GitHub Action. It uses run_action.py to execute the detection and entrypoint.sh to manage the Git environment during a Pull Request.

This diagram illustrates how the GitHub Action wrapper interacts with the core detection script.
Sources: action.yml:1-60, run_action.py:115-160, entrypoint.sh:1-25

CLI Usage

The tool provides a command-line interface named duplicate-code-detection.

Argument	Description	Default
-d, --directories	List of directories to analyze	N/A
-f, --files	Specific files to compare	N/A
-t, --fail-threshold	Similarity percentage to trigger failure	100
--ignore-threshold	Similarity percentage to ignore in output	0
-j, --json	Output results in JSON format	False

Sources: duplicate_code_detection.py:130-190, README.md:67-80

Execution Results

The tool returns one of three primary status codes via the ReturnCode Enum:

1. SUCCESS (0): Analysis completed, and no duplications exceeded the fail_threshold.
2. BAD_INPUT (1): Invalid paths or insufficient files (minimum 2) provided.
3. THRESHOLD_EXCEEDED (2): At least one duplication pair exceeded the configured fail_threshold.

Sources: duplicate_code_detection.py:46-50, core.py:17-21

Summary

The Duplicate Code Detection Tool provides a robust mechanism for maintaining codebases by identifying redundant logic. Through its integration with Flext projects and CI/CD environments, it enables automated enforcement of code quality standards using configurable similarity metrics and flexible reporting formats.

Sources: README.md:200-210, examples/flext_integration_example.py:130-150