Directory and Project Analysis
Relevant source files
The following files were used as context for generating this wiki page:
Directory and Project Analysis refers to the systematic process of scanning file systems, identifying source code files, and evaluating them for content similarity. This system is designed to handle both localized file sets and large-scale project structures, facilitating quality assurance and code maintenance within Flext projects. The analysis supports multiple programming languages, handles exclusion patterns, and provides structured results for further integration into CI/CD pipelines.
The core logic resides in the DuplicateCodeDetector class and the run function, which orchestrate the discovery of files across directories and the subsequent similarity computation using Natural Language Processing (NLP) techniques.
Sources: core.py:126-140, duplicate_code_detection.py:155-188, README.md:9-16
Analysis Architecture and Logic
The analysis workflow transitions from raw file discovery to tokenization and similarity indexing. The process involves identifying files based on extensions, filtering out ignored paths, and utilizing TF-IDF (Term Frequency-Inverse Document Frequency) models to determine the probability of duplication.
Data Flow for Project Analysis
The following diagram illustrates how the system processes a project directory into a finalized similarity report.
The system uses os.walk to recursively traverse directories and word_tokenize from nltk to break code into consumable parts for the gensim similarity engine.
Sources: duplicate_code_detection.py:48-60, duplicate_code_detection.py:233-255, core.py:142-168
Directory Discovery and Filtering
The tool identifies relevant files within a directory based on a list of supported source code extensions. It supports sophisticated filtering to ensure that third-party libraries (e.g., node_modules) or generated files are not included in the analysis.
Discovery Parameters
| Parameter | Description | Source |
|---|---|---|
directories | List of root paths to begin recursive scanning. | duplicate_code_detection.py:161 |
file_extensions | Specific extensions to include (e.g., .py, .js, .cpp). | duplicate_code_detection.py:34 |
ignore_directories | Sub-directories to be skipped during the walk. | duplicate_code_detection.py:168 |
project_root_dir | A base path used to convert absolute file paths into relative ones for reporting. | duplicate_code_detection.py:192-198 |
The get_all_source_code_from_directory function is the primary mechanism for this phase:
def get_all_source_code_from_directory(directory, file_extensions):
source_code_files = list()
for dirpath, _, filenames in os.walk(directory):
for name in filenames:
_, file_extension = os.path.splitext(name)
if file_extension[1:] in file_extensions:
filename = os.path.join(dirpath, name)
source_code_files.append(filename)
return source_code_files
Sources: duplicate_code_detection.py:48-60
Content Processing
Once files are identified, the system reads their content. For Python files, the tool can optionally remove comments and docstrings using the ast and astor libraries to focus analysis purely on functional code logic.
Pre-processing Sequence
The sequence diagram below shows how a single file's content is transformed before similarity evaluation.
Results and Reporting
The analysis produces a structured hierarchy of results, ranging from individual file similarities to project-wide summaries.
Result Objects
- DetectionResult: The top-level container holding the success status and a list of reports.
- SimilarityReport: Detailed data for a specific file, including its line count (LoC) and its similarity to other files.
- FileSimilarity: The specific relationship between a source and target file, including the similarity percentage.
Sources: core.py:27-80
Failure Conditions
The system evaluates results against a fail_threshold. If the similarity between any two files exceeds this value, the ReturnCode is set to THRESHOLD_EXCEEDED (value 2), allowing CI/CD pipelines to fail the build.
Sources: duplicate_code_detection.py:41, run_action.py:139-144
Project Integration Example
Integration for project-wide analysis is typically handled via the analyze_project method, which automatically handles relative path formatting and directory exclusion.
detector = DuplicateCodeDetector(
fail_threshold=80,
ignore_threshold=15,
file_extensions=["py"],
only_code=True
)
result = detector.analyze_project(
project_path=".",
exclude_patterns=["tests/", "docs/", ".git/"]
)
Sources: README.md:43-60, core.py:284-307
Conclusion
Directory and Project Analysis provides a robust framework for identifying redundant code across complex file structures. By combining directory traversal, AST-based pre-processing, and NLP-driven similarity detection, the tool ensures that codebases remain maintainable and free of excessive duplication.