Project & Language Detection

Relevant source files

The following files were used as context for generating this wiki page:

• crates/mcb-validate/src/utils/source.rs
• crates/mcb-validate/tests/unit/filters/language_detector_tests.rs
• crates/mcb-providers/Cargo.toml
• config/mcb-validate-internal.toml
• crates/mcb-domain/src/ports/validation.rs (Referenced via domain dependency in mcb-providers/Cargo.toml)
• README.md

Project and Language Detection is a core capability of the Memory Context Browser (MCB) used to classify codebases, identify programming languages, and manage repository-level context. This system enables MCB to apply language-specific parsing, provide accurate syntax highlighting, and facilitate AST-aware analysis across 14+ supported languages.

The detection logic is distributed across providers for project manifest parsing (e.g., Cargo, Python) and validation filters for file-level language identification. It integrates closely with the indexing engine to ensure that the semantic search and persistent memory features are grounded in the correct technical context of the project being browsed.

Sources: README.md:16-24, crates/mcb-providers/Cargo.toml:78-85

Language Identification Logic

MCB utilizes a LanguageDetector component to identify programming languages through a multi-tiered heuristic approach involving file extensions, specific filenames, and content analysis (shebangs).

Detection Tiers

The LanguageDetector evaluates files in the following order of priority:

1. Filename Mapping: Matches specific configuration or build files (e.g., Dockerfile, Makefile).
2. Extension Mapping: Maps standard file extensions to internal LanguageId values (e.g., .rs to Rust, .py to Python).
3. Content Detection: Inspects the first line of a file for shebangs (e.g., #!/bin/bash) to identify scripts without standard extensions.

Supported Languages and Mappings

MCB supports 20 distinct language identifiers for validation and search.

Detection Method	Input Examples	Resulting LanguageId
Extension	.rs, .py, .js, .tsx, .go, .rb, .yaml, .toml, .json, .md, .html, .css, .sql, .proto	Rust, Python, JavaScript, TypeScript, Go, Ruby, Yaml, Toml, Json, Markdown, Html, Css, Sql, Protobuf
Filename	Dockerfile, Makefile, GNUmakefile, CMakeLists.txt	Dockerfile, Makefile, Cpp
Shebang	#!/usr/bin/env python, #!/bin/bash, #!/usr/bin/env ruby	Python, Shell, Ruby

Sources: crates/mcb-validate/tests/unit/filters/language_detector_tests.rs:13-162

Project Detection Architecture

Project detection identifies the type of software project by parsing manifests and repository structures. This is primarily handled in mcb-providers and used during the indexing phase to determine how a repository should be treated.

Manifest Parsing

The system includes specialized parsers for various ecosystems to extract metadata such as package names, versions, and dependencies.

• Rust: Uses cargo_toml to parse Cargo.toml.
• Python: Uses toml to parse pyproject.toml.
• General: Uses quick-xml for XML-based manifests.

Directory Scanning

The mcb-validate crate provides utilities to traverse project structures. The for_each_rust_file helper specifically iterates over source directories defined in the ValidationConfig, filtering for specific languages and skipping non-source areas like tests/fixtures/ or target/.

Implementation Details

Source Analysis Utilities

In crates/mcb-validate/src/utils/source.rs, the system implements logic to refine language detection during analysis. For example, when identifying Rust source code, it excludes #[cfg(test)] regions and comments to focus on meaningful implementation.

pub fn source_lines(content: &str) -> Vec<(usize, &str)> {
    let mut result = Vec::new();
    let mut in_test_module = false;
    for (idx, line) in content.lines().enumerate() {
        let trimmed = line.trim();
        if trimmed.starts_with(COMMENT_PREFIX) { continue; }
        if trimmed.contains(CFG_TEST_MARKER) {
            in_test_module = true;
            continue;
        }
        if in_test_module { continue; }
        result.push((idx + 1, trimmed));
    }
    result
}

Sources: crates/mcb-validate/src/utils/source.rs:64-84

Configuration-Driven Detection

Project boundaries and internal dependency prefixes are defined in configuration files. mcb-validate-internal.toml specifies the project_prefix (e.g., "mcb") and internal_dep_prefix (e.g., "mcb-") which aids the detector in distinguishing between internal workspace crates and external third-party dependencies.

Sources: config/mcb-validate-internal.toml:6-10

Conclusion

Project and Language Detection in MCB provides the structural awareness necessary for semantic search and architecture validation. By combining file-level language identification with project-level manifest parsing, MCB accurately builds a context model of the codebase, enabling features like AST-aware chunking and cross-session persistent memory.