Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• core.py
• duplicate_code_detection.py
• run_action.py
• README.md
• action.yml
• entrypoint.sh

The Duplicate Code Detection Tool is a Python-based utility designed to identify similarities and duplications across source code files. It functions as both a standalone CLI tool and a library, with specialized integration for GitHub Actions and Flext projects. The system leverages Natural Language Processing (NLP) techniques to compute similarity scores between documents, enabling quality gates in CI/CD pipelines.

Sources: README.md:1-15, core.py:1-8

System Components

The architecture is divided into three primary layers: the Core Analysis Engine, the Library Wrapper, and the Integration Layer.

1. Core Analysis Engine

This layer handles the heavy lifting of file discovery, text preprocessing, and similarity calculation. It uses gensim for document similarity and nltk for tokenization.

• File Discovery: Recursively scans directories for files matching specified extensions.
• Preprocessing: Optionally removes Python comments and docstrings using the ast and astor modules.
• Similarity Logic: Implements a TF-IDF (Term Frequency-Inverse Document Frequency) model to convert source code into vectors and calculate similarity scores.

Sources: duplicate_code_detection.py:46-56, duplicate_code_detection.py:68-100, duplicate_code_detection.py:233-242

2. Library Wrapper (core.py)

Provides an object-oriented interface for developers. It introduces high-level abstractions like DuplicateCodeDetector to manage analysis sessions and DetectionResult to encapsulate findings.

Sources: core.py:114-131, core.py:69-111

3. Integration Layer

Facilitates use within GitHub Actions. run_action.py acts as a bridge, consuming environment variables, executing the analysis, and posting results back to GitHub Pull Requests as Markdown comments.

Sources: run_action.py:133-162, action.yml:1-60

Data Flow Architecture

The following diagram illustrates the flow of data from input parameters to the final similarity report.

The workflow begins by resolving target files, processing their content based on configuration, and utilizing a TF-IDF model to generate a similarity matrix.
Sources: duplicate_code_detection.py:166-260, core.py:157-190

GitHub Action Execution Flow

When triggered as a GitHub Action, the tool follows a specific sequence to interact with the repository environment.

The execution environment uses a Docker-based runner that clones the repository and checks out the specific Pull Request branch before initiating analysis.
Sources: entrypoint.sh:4-22, run_action.py:183-228

Key Classes and Data Structures

The system uses specialized data classes to maintain structured information about code duplications.

Class	Description	Key Attributes
FileSimilarity	Represents a match between two files.	source_file, target_file, similarity_percentage, source_loc
SimilarityReport	Collection of matches for a specific file.	file_path, similarities (List), loc_count
DetectionResult	The final output object of an analysis run.	return_code, reports (List), raw_data
DuplicateCodeDetector	Main entry point for library usage.	fail_threshold, ignore_threshold, only_code

Sources: core.py:27-111

Configuration and Exit Logic

The tool employs a ReturnCode enumeration to signal the outcome of the analysis to the calling environment.

class ReturnCode(Enum):
    SUCCESS = 0             # Analysis finished, threshold not exceeded
    BAD_INPUT = 1           # Invalid directories or file paths provided
    THRESHOLD_EXCEEDED = 2  # A file similarity surpassed the fail_threshold

Sources: duplicate_code_detection.py:33-36

Input Parameters (Action)

Input	Description	Default
fail_above	Max allowed similarity before failing (1-100).	100
ignore_below	Min similarity to report.	10
only_code	Strip comments/docstrings from Python files.	false
one_comment	Update existing PR comment instead of creating new ones.	false

Sources: action.yml:10-48

Conclusion

The Duplicate Code Detection Tool's architecture effectively separates its core NLP-based engine from its consumption interfaces. By utilizing TF-IDF for similarity detection and providing both a structured library and a robust GitHub Action implementation, it serves as a flexible component for maintaining code quality and identifying technical debt within software projects.

Sources: README.md:145-155, core.py:246-270