File Extension Support

The Duplicate Code Detection Tool provides a flexible mechanism for filtering and identifying source code files based on their extensions. This functionality ensures that the analysis engine focuses only on relevant text-based source files, preventing the inclusion of binary data or unsupported formats that could compromise the accuracy of the similarity metrics.

The system supports a broad range of programming languages by default, including Python, JavaScript, TypeScript, Java, and C-style languages. Users can override these defaults through library configurations, CLI arguments, or GitHub Action inputs to tailor the detection process to specific project requirements.
Sources: README.md:120-126, duplicate_code_detection.py:27

Extension Configuration

The tool maintains two primary sets of supported file extensions depending on the interface used (CLI/Core vs. GitHub Action). These extensions define which files are gathered during directory traversal.

Default Supported Extensions

Sources: duplicate_code_detection.py:27, core.py:122-130, action.yml:21-23

Architecture of File Discovery

The file discovery process uses the get_all_source_code_from_directory function. This function performs a recursive walk through the specified directory, splitting each filename to extract its extension and comparing it against the allowed list.

The diagram illustrates how the file_extensions list acts as a filter during the directory crawling phase.
Sources: duplicate_code_detection.py:53-64

Implementation Details

Library-Level Support

In the core.py module, the DuplicateCodeDetector class initializes the allowed extensions in its constructor. If no list is provided by the user, it defaults to a modern set including web-focused languages like JavaScript and TypeScript.

# From core.py
self.file_extensions = file_extensions or [
    "py",
    "js",
    "ts",
    "java",
    "cpp",
    "c",
    "h",
]

Sources: core.py:122-130

Action Runner Integration

The run_action.py script receives extensions as a comma-separated string from the GitHub Action environment variable INPUT_FILE_EXTENSIONS. It uses a utility function split_and_trim to convert this string into a Python list before passing it to the main analysis engine.
Sources: run_action.py:133, run_action.py:141

Special Handling for Python Files

While the tool performs generic tokenization for most extensions, the .py extension triggers an optional pre-processing step. When the only_code flag is enabled, the tool utilizes the ast and astor libraries to strip comments and docstrings specifically from Python source files.

The workflow shows how the file extension dictates whether advanced AST-based parsing is applied before similarity analysis.
Sources: duplicate_code_detection.py:223-225, duplicate_code_detection.py:70-101

Configuration via Interfaces

Users can customize file extension support across all supported interfaces:

Command Line Interface (CLI)

Using the --file-extensions flag allows users to pass a space-separated list of extensions.

duplicate-code-detection -d ./src --file-extensions py js ts

Sources: duplicate_code_detection.py:148-152

GitHub Action

Extensions are defined in the with block of the action workflow as a comma-separated list.

- uses: flext-sh/duplicate-code-detection-tool@v2
  with:
    file_extensions: 'py, go, rs'

Sources: action.yml:21-23

Core Library

When instantiating the DuplicateCodeDetector, the file_extensions parameter accepts a list of strings.
Sources: core.py:112, README.md:104

Summary

File extension support is a foundational filtering layer within the Duplicate Code Detection Tool. By utilizing the os.path.splitext method during directory traversal and maintaining configurable defaults across different interfaces, the tool ensures that only relevant source code is analyzed. Furthermore, the system leverages specific extensions like .py to enable advanced features such as comment removal via AST parsing.
Sources: duplicate_code_detection.py:53-64, core.py:122-130, duplicate_code_detection.py:223-225