Code Sanitization

Relevant source files

The following files were used as context for generating this wiki page:

• core.py
• duplicate_code_detection.py
• README.md
• action.yml
• examples/flext_integration_example.py
• run_action.py

Code Sanitization within the Duplicate Code Detection Tool refers to the process of preparing and cleaning source code files before they are analyzed for similarities. Its primary purpose is to ensure that the detection logic focuses on the actual logic of the code rather than superficial elements such as comments, docstrings, or formatting. This process is critical for reducing false positives and identifying meaningful duplication across different files or projects.

The sanitization module is primarily implemented through specific flags in the DuplicateCodeDetector class and the underlying run function, specifically utilizing the only_code parameter to trigger the removal of non-executable elements.

Sources: README.md:14-18, core.py:126-140, duplicate_code_detection.py:90-95

Sanitization Workflow

The sanitization process is an optional step that occurs during the file reading phase of the analysis. When enabled, the tool processes Python source files to strip away metadata and documentation that does not contribute to functional logic.

The flow shows how the only_code flag intercepts the standard file reading process for Python files.
Sources: duplicate_code_detection.py:270-282, core.py:157-160

Key Components and Logic

Comment and Docstring Removal

The core of the sanitization logic resides in the remove_comments_and_docstrings function. This function uses Python's Abstract Syntax Tree (ast) to traverse the code structure and identify nodes that represent documentation.

Element	Handling Logic
Docstrings	Identified as ast.Str values within the first ast.Expr of a module, class, or function body. These are sliced out of the node body.
Comments	Removed during the parsing and re-generation phase, as ast does not preserve standard comments.
Code Structure	The astor library is used to convert the modified AST back into a clean source code string.

Sources: duplicate_code_detection.py:90-120, requirements.txt:3

AST Transformation

The transformation identifies specific node types including FunctionDef, ClassDef, AsyncFunctionDef, and Module. It checks if the first item in the body is an expression containing a string (the docstring) and removes it if present.

# From duplicate_code_detection.py:90-120
def remove_comments_and_docstrings(source_code: str) -> str:
    parsed = ast.parse(source_code)
    for node in ast.walk(parsed):
        if not isinstance(
            node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef, ast.Module)
        ):
            continue
        # ... validation checks ...
        if not hasattr(node.body[0], "value") or not isinstance(
            node.body[0].value, ast.Str
        ):
            continue
        node.body = node.body[1:]
    return astor.to_source(parsed)

Sources: duplicate_code_detection.py:90-120

Configuration and Integration

Sanitization is controlled via the only_code parameter across various entry points of the tool.

Configuration Options

Parameter	Type	Default	Description
only_code	boolean	False	When True, removes comments and docstrings from Python source code before analysis.
file_extensions	list/str	py, js, ts...	Extensions to include; sanitization logic currently targets .py files specifically.

Sources: action.yml:41-45, duplicate_code_detection.py:192-196, core.py:139

Library Integration Example

Developers can invoke sanitization by passing only_code=True to the DuplicateCodeDetector constructor or convenience functions like analyze_project_duplicates.

from duplicate_code_tool import DuplicateCodeDetector

detector = DuplicateCodeDetector(
    only_code=True,         # Enable sanitization
    file_extensions=["py"]
)

Sources: examples/flext_integration_example.py:61-68, README.md:43-49

GitHub Action Implementation

In the context of a CI/CD pipeline, the run_action.py script captures the INPUT_ONLY_CODE environment variable from action.yml and passes it to the underlying duplicate_code_detection.run function.

Conclusion

Code Sanitization is a vital preprocessing step that enhances the accuracy of the Duplicate Code Detection Tool. By stripping non-functional text and normalizing the Abstract Syntax Tree, the tool ensures that the similarity reports reflect actual logic duplication, which is essential for effective refactoring and code quality maintenance.

Sources: README.md:14-18, examples/flext_integration_example.py:202-208