Batch File Operations
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Batch File Operations in CodeBot provide a high-performance mechanism for performing concurrent actions on multiple code files. This system allows users to select groups of files based on repository tags, file types, or manual selection and apply complex operations such as static code analysis, validation, and multi-file export. The batch system is designed to handle long-running tasks asynchronously, providing real-time progress updates and status monitoring without blocking the main bot interaction loop.
The core of this feature is the BatchProcessor, which manages the lifecycle of a BatchJob from creation to completion. It leverages a thread pool to execute file-specific operations in parallel, ensuring efficient resource utilization even when dealing with large projects or complex validation tasks.
Sources: batch_processor.py:1-60, conversation_handlers.py:1262-1275
Core Components and Data Structures
BatchJob Data Model
The system tracks every batch operation using the BatchJob dataclass. This structure stores the operational context, including the user responsible, the specific files targeted, and the current state of execution.
| Field | Type | Description |
|---|---|---|
job_id | str | Unique identifier for the batch task. |
user_id | int | The ID of the user who initiated the job. |
operation | str | Type of operation (e.g., "analyze", "validate", "export"). |
status | str | Current state: pending, running, completed, or failed. |
progress | int | Number of files processed so far. |
total | int | Total number of files in the batch. |
results | Dict | Mapping of filenames to their individual results or errors. |
error_message | str | Global error message if the entire batch fails. |
Sources: batch_processor.py:22-42
BatchProcessor Architecture
The BatchProcessor class manages active jobs and handles thread orchestration. It is configured with concurrency limits to prevent system exhaustion while maintaining a realistic processing feel in the UI.
The diagram shows the relationship between the processor and the individual job instances it tracks.
Sources: batch_processor.py:44-70
Operation Lifecycle and Flow
Batch Processing Sequence
When a batch operation is triggered, the system follows a specific sequence: job creation, background task scheduling, parallel execution, and status polling.
The sequence illustrates the asynchronous nature of batch operations where the user receives an ID immediately while processing continues in the background.
Sources: batch_processor.py:84-145, conversation_handlers.py:1288-1322
Key Batch Features
Static Code Analysis
The analyze_files_batch function performs deep inspection of code. It retrieves the latest version of every file in the batch and applies code_processor.analyze_code to calculate metrics like line counts and language-specific characteristics.
Sources: batch_processor.py:182-212
Multi-File Validation
The validation engine performs both syntax checks and advanced linter-based inspections (e.g., flake8, mypy, bandit). It utilizes temporary directories to run these tools against the files in the batch.
- Python Checks: Includes
flake8,mypy,bandit,black, andpylint. - Web Checks: Includes
eslintandprettierfor JS/TS. - Security: A
_secrets_scanfunction uses regex patterns to detect AWS keys, GitHub tokens, and private keys.
Sources: batch_processor.py:214-360
Job Monitoring and Management
The bot interface provides real-time monitoring of batch tasks.
| Function | Logic |
|---|---|
format_job_summary | Generates a visual progress bar (e.g., [███░░░░░░░]) and completion stats. |
cleanup_old_jobs | Automatically removes jobs older than 24 hours to save memory. |
cancel_job | Marks a running job as failed with a "canceled by user" message. |
_auto_update_batch_status | An asynchronous loop that updates the bot message every 2 seconds until completion. |
Sources: batch_processor.py:381-447, conversation_handlers.py:1550-1579
Batch Integration with UI
Users interact with the batch system through the batch_menu. The UI allows filtering files by repository (repo:owner/name), file size, or specific file names before starting the batch.
This flow represents the navigation paths available to the user in the bot interface for initiating batch tasks.
Sources: conversation_handlers.py:1416-1465
Conclusion
Batch File Operations represent a critical productivity feature in CodeBot, enabling scalable management of large codebases. By decoupling file processing from the user interface via a background BatchProcessor and providing comprehensive validation and analysis tools, the system allows for professional-grade code auditing directly within the Telegram interface. The robust lifecycle management, including auto-cleanup and progress tracking, ensures the system remains stable and responsive under high load.
Sources: batch_processor.py:449, GUIDES/ARCHITECTURE.md:200-220