Code Parsing & Language Detection

Relevant source files

The following files were used as context for generating this wiki page:

• src/domain/services/code_normalizer.py
• src/domain/services/language_detector.py
• repo_analyzer.py
• services/snippet_library_service.py
• GUIDES/ARCHITECTURE_ANALYSIS_PART1.md
• FEATURE_SUGGESTIONS/README.md

The Code Parsing & Language Detection system within CodeBot is a critical domain-level infrastructure responsible for identifying programming languages and preparing code snippets for storage and analysis. It ensures that raw user input is transformed into a standardized format, labeled with the correct syntax metadata, and evaluated for structural integrity. This module bridges the gap between raw user messages/files and the persistence layer, enabling advanced features like syntax highlighting and automated code review.

This system is integrated into several high-level workflows, including the Snippet Library Service for community contributions and the Repository Analyzer for automated GitHub project inspections.

Core Domain Services

The architecture follows a layered approach where parsing and detection logic reside in the src/domain/services layer. This ensures that business rules regarding code handling are decoupled from specific communication interfaces like Telegram or the WebApp.

Language Detection

The LanguageDetector service is the primary authority for identifying programming languages. It utilizes a combination of file extensions, content analysis, and metadata provided by external APIs (e.g., Pygments or GitHub API) to determine the most likely language for a given snippet.

The system supports a wide range of extensions, including .py, .js, .ts, .java, .cpp, .go, and .rs.
Sources: repo_analyzer.py:46-50, src/domain/services/language_detector.py

Code Normalization

Normalization is a prerequisite for consistency across the platform. The CodeNormalizer (previously residing in utils.py) is now part of the domain layer to centralize rules for code sanitization. This includes stripping trailing whitespace, standardizing line endings, and ensuring character encoding is consistent before the code reaches the database.

Feature	Description	File Source
Sanitization	Truncates text to maximum allowed lengths (e.g., 180 for titles, 1000 for descriptions).	services/snippet_library_service.py:17-25
Encoding	Standardizes characters to UTF-8 for cross-platform compatibility.	src/domain/services/code_normalizer.py
Security Validation	Checks for dangerous patterns like eval(), exec(), or rm -rf.	services/snippet_library_service.py:537-565

Repository Analysis and Structural Parsing

CodeBot extends simple snippet parsing into full repository analysis through the RepoAnalyzer class. This component parses the structure of GitHub repositories to identify key project components and code quality metrics.

Analysis Workflow

The RepoAnalyzer performs a multi-step parsing process:

1. URL Parsing: Extracts the owner and repository name from GitHub URLs.
2. Structural Scanning: Traverses the directory structure to identify configuration files (e.g., package.json, requirements.txt).
3. Code Feature Extraction: Identifies large files and long functions (e.g., functions exceeding 50 lines).

Structural Quality Metrics

During parsing, the system identifies "Code Smells" based on structural thresholds:

• Large Files: Files exceeding 500 lines.
• Long Functions: Functions (Python/JS/TS) exceeding 50 lines.
• Documentation Quality: Parsed based on the presence and content of README.md.
  Sources: repo_analyzer.py:40-44, repo_analyzer.py:127-140

Snippet Library Architecture

The SnippetLibraryService utilizes the parsing and detection layer to manage community-submitted code. Before a snippet is persisted, it undergoes validation and language verification.

Submission Process

When a user submits code, the service coordinates between the normalizer and the repository layer.

Validation Rules

The service enforces specific constraints during parsing:

• Title Length: Minimum 3 characters, maximum 180 characters.
• Description Length: Minimum 4 characters.
• Required Fields: Code and Language must not be empty.
  Sources: services/snippet_library_service.py:112-118

Technical Implementation Details

Dependency Extraction

The system includes specialized parsers for various dependency manifests. This is critical for generating improvement suggestions regarding unpinned versions.

# Extraction logic for requirements.txt in repo_analyzer.py
def _extract_dependencies(self, filename: str, content: str) -> List[Dict[str, str]]:
    if filename == 'requirements.txt':
        for line in content.split('\n'):
            # Extracts name and version using regex [>=<~!]
            parts = re.split(r'[>=<~!]', line.strip())
            # Result: {'name': 'requests', 'type': 'python', 'version': '2.28.1'}

Sources: repo_analyzer.py:286-302

Quality Scoring Model

The results of code parsing are aggregated into a numerical quality score (0-100).

Component	Weight	Criteria
Base Files	40 points	README (15-25), License (10), Gitignore (5)
CI/CD & Tests	30 points	Workflow files (15), Test directories (15)
Code Quality	30 points	Absence of large files (10), Absence of long functions (10)
Popularity	5 points	Stars > 100 (Bonus)
Sources: repo_analyzer.py:338-375

Conclusion

The Code Parsing & Language Detection system provides the foundational intelligence for CodeBot. By centralizing normalization and detection in the domain layer and implementing specialized structural parsers in the RepoAnalyzer, the project maintains high standards for code quality and metadata accuracy. This architecture supports the project's goal of becoming a production-grade enterprise solution for code management.
Sources: code_review_2025_10/CODE_REVIEW_SUMMARY.md