Batch Processing

Relevant source files

The following files were used as context for generating this wiki page:

• examples/flext_integration_example.py
• README.md
• duplicate_code_detection.py
• core.py
• run_action.py

Batch Processing in the Duplicate Code Detection Tool refers to the systematic analysis of multiple source code files or entire directory structures to identify similarities in a single execution flow. This capability is designed to support large-scale code quality audits, CI/CD pipeline integration, and multi-project analysis by processing groups of files rather than individual pairs.

The system utilizes a vector-space model (Gensim) to convert entire collections of source documents into a searchable index, allowing the tool to calculate similarity percentages across all files in the batch efficiently.

Sources: README.md:183-185, duplicate_code_detection.py:270-280

Architecture and Data Flow

The batch processing logic follows a structured pipeline: discovery, pre-processing, indexing, and comparison. The system first discovers all relevant files based on user-defined extensions and exclusion patterns. It then reads these files into memory, optionally stripping non-functional code elements like comments or docstrings for a more accurate logic-based comparison.

The diagram above illustrates the transformation of raw source files into a searchable similarity index during a batch run.

Processing Components

1. File Discovery: The get_all_source_code_from_directory function recursively walks through directory paths to find files matching supported extensions like .py, .js, .ts, .java, etc.
2. Logic Extraction: If only_code is enabled, the tool uses ast and astor to parse Python files and remove docstrings and comments.
3. Indexing: The gensim.similarities.Similarity class creates a sharded index on disk (often in tempfile.gettempdir()) to handle the batch comparison of the corpus.

Sources: duplicate_code_detection.py:53-64, duplicate_code_detection.py:73-100, duplicate_code_detection.py:270-285, core.py:126-140

Configuration for Large Batches

When processing large projects, several parameters control the behavior and performance of the batch run. These can be configured via the DuplicateCodeDetector class or CLI arguments.

Parameter	Type	Default	Description
directories	list	N/A	List of root paths to scan for source files.
ignore_directories	list	[]	Paths to exclude from the batch (e.g., tests/, node_modules/).
file_extensions	list	[py, js, ts, java, cpp, c, h]	Filters the batch to specific languages.
ignore_threshold	int	15	Minimum similarity % to include in the batch report.
fail_threshold	int	100	Triggers a non-zero exit code if any similarity exceeds this.

Sources: core.py:100-124, duplicate_code_detection.py:151-205, README.md:215-225

Multi-Project Analysis

The tool supports "Batch Analysis" of multiple distinct projects. This is implemented by iterating through project paths and invoking the analyze_project_duplicates function for each. This allows developers to maintain a consistent quality gate across a modular architecture, such as the Flext ecosystem.

def analyze_multiple_projects(project_paths: list[str]):
    results = {}
    for project_path in project_paths:
        result = analyze_project_duplicates(
            project_path=project_path,
            threshold=20.0,
            exclude_patterns=["tests/", "docs/", ".git/"]
        )
        results[project_path] = result
    return results

Sources: README.md:183-205, examples/flext_integration_example.py:90-110

Reporting and Output Formats

Batch results are encapsulated in a DetectionResult object, which aggregates individual SimilarityReport instances for every file in the batch.

Result Structure

• DetectionResult: The root container for the batch run, providing a success status and a collection of reports.
• SimilarityReport: Contains the analysis for a single source file, including its line count (LoC) and a list of FileSimilarity objects representing matches with other files in the batch.

This class diagram shows the hierarchical relationship of data objects returned after a batch processing task.

Sources: core.py:24-95, duplicate_code_detection.py:348-385

Conclusion

Batch processing is the core functional mode of the Duplicate Code Detection Tool, enabling high-throughput analysis of codebases. By leveraging the DuplicateCodeDetector class or the CLI, users can integrate complex code similarity checks into their development workflows, ensuring consistency across large or multiple projects while filtering out irrelevant files and directories.

Sources: README.md:10-20, core.py:98-105