Core Detection Engine

Relevant source files

The following files were used as context for generating this wiki page:

• core.py
• duplicate_code_detection.py
• run_action.py
• README.md
• examples/flext_integration_example.py

The Core Detection Engine is the central processing component of the Duplicate Code Detection Tool. It provides the algorithmic logic and high-level abstractions necessary to identify similarities between source code files across various programming languages. The engine is designed to operate as a standalone CLI tool, a GitHub Action, or a Python library for integration into third-party projects.

The engine leverages natural language processing techniques, specifically TF-IDF (Term Frequency-Inverse Document Frequency) and cosine similarity via the gensim library, to compare tokenized source code. It supports filtering mechanisms such as comment removal and specific file extension targeting to ensure analysis focus remains on functional code rather than documentation or boilerplate.

Sources: README.md:1-15, duplicate_code_detection.py:10-15, core.py:1-10

Architecture and Components

The engine is structured into a low-level functional layer and a high-level object-oriented wrapper.

Processing Pipeline

The detection process follows a linear pipeline: file discovery, content preprocessing, tokenization, model generation, and similarity calculation.

The pipeline begins by identifying valid source files, optionally stripping non-code elements via AST parsing, and then applying numerical vectorization to compute similarity scores.
Sources: duplicate_code_detection.py:214-275, core.py:127-145

High-Level Abstractions

The core.py module introduces a class-based interface to manage detection tasks and results.

Class	Purpose	Key Attributes/Methods
DuplicateCodeDetector	Primary interface for analysis.	detect_in_directories(), analyze_project()
DetectionResult	Container for the outcome of an analysis.	is_success(), get_critical_duplications()
SimilarityReport	Detailed findings for a specific file.	file_path, similarities, loc_count
FileSimilarity	Data object for a file pair similarity.	similarity_percentage, source_loc, target_loc

Sources: core.py:22-120

Detection Logic and Preprocessing

Content Transformation

To improve accuracy, the engine can preprocess Python files by removing comments and docstrings. This is achieved using the ast and astor libraries to walk the Abstract Syntax Tree and prune expression nodes containing string literals.

def remove_comments_and_docstrings(source_code: str) -> str:
    parsed = ast.parse(source_code)
    for node in ast.walk(parsed):
        # ... logic to identify and remove docstring nodes ...
    source_code_clean = astor.to_source(parsed)
    return source_code_clean

Sources: duplicate_code_detection.py:65-95

Similarity Computation

The engine utilizes gensim for document similarity. It tokenizes source code into lowercase words, builds a dictionary of all terms, and converts files into "Bag of Words" (BoW) representations. A TF-IDF model is then applied to weight terms before calculating cosine similarity.

Configuration and Parameters

The engine's behavior is governed by several threshold and filtering parameters used across CLI, Library, and Action interfaces.

Parameter	Type	Description	Default
fail_threshold	int	Similarity % above which the tool returns a failure code.	100
ignore_threshold	int	Similarity % below which results are excluded from reports.	0
file_extensions	list	List of file suffixes to include (e.g., py, cpp).	[h, c, cpp, cc, java, py, cs]
only_code	bool	If True, removes Python comments/docstrings before analysis.	False
show_loc	bool	If True, calculates and reports Lines of Code for analyzed files.	False

Sources: duplicate_code_detection.py:112-154, action.yml:8-45, core.py:127-145

Result Handling and Integration

The engine returns structured data that can be consumed in multiple formats. The run() function in duplicate_code_detection.py returns a tuple containing a ReturnCode and a dictionary of similarities.

Return Codes

• SUCCESS (0): Detection completed; no thresholds exceeded.
• BAD_INPUT (1): Invalid directories, files, or arguments provided.
• THRESHOLD_EXCEEDED (2): Similarity between files surpassed the fail_threshold.

Sources: duplicate_code_detection.py:32-35, core.py:15-20

Reporting Outputs

1. JSON: A nested dictionary structure mapping source files to their similar counterparts and percentage scores.
2. CSV: A flat file containing columns for "File A", "File B", and "Similarity".
3. Markdown: Used primarily for GitHub Action comments, including collapsible sections and tables.

Sources: run_action.py:15-95, duplicate_code_detection.py:345-385

Conclusion

The Core Detection Engine provides a robust framework for static code analysis focused on duplication. By combining AST-based preprocessing with established NLP similarity algorithms, it offers a flexible solution for maintaining code quality in CI/CD pipelines and local development environments. Its modular design allows it to scale from simple file comparisons to large-scale project audits.

Sources: README.md:200-215, examples/flext_integration_example.py:150-165