Concept Extraction Logic
Relevant source files
The following files were used as context for generating this wiki page:
The Concept Extraction Logic is a core subsystem of CodeScholar designed to bridge the gap between source code and academic research. Its primary purpose is to analyze source code to identify algorithms, machine learning techniques, data structures, and design patterns. These identified "concepts" are then used to generate optimized search queries for the CodeScholar paper library, allowing developers to discover the research foundations of their implementation.
The system employs a multi-tiered analysis strategy, ranging from Abstract Syntax Tree (AST) parsing for Python to regex-based pattern matching for other languages. This extraction serves as the entry point for the analyze_code and link_papers_to_code tools provided by the CodeScholar MCP server.
Sources: scripts/analyzers/concept_extractor.py:1-12, server.js:15-25
System Architecture and Data Flow
The extraction process begins when a user provides a file path or directory through the MCP server. The logic flows from raw source code through language-specific analyzers to a structured set of extracted concepts.
High-Level Analysis Flow
The following diagram illustrates the flow from the initial request to the generation of search queries:
The system distinguishes between formal code structures and informal text within the code to maximize detection coverage.
Sources: scripts/analyzers/concept_extractor.py:344-428, server.js:475-500
Core Components
Python AST Analyzer
The PythonASTAnalyzer class inherits from ast.NodeVisitor to perform deep inspection of Python source code. It targets specific nodes to identify academic concepts based on imports, function names, and class definitions.
| Targeted Node | Extraction Logic | Confidence Impact |
|---|---|---|
Import / ImportFrom | Maps library names (e.g., sklearn, torch) to concepts. | High (0.8 - 0.95) |
FunctionDef | Matches function names against CODE_PATTERNS. | High (0.9) |
ClassDef | Matches class names and base classes against patterns. | High (0.9) |
Sources: scripts/analyzers/concept_extractor.py:149-210
Regex-Based Analysis
For languages like JavaScript and TypeScript, the system utilizes the analyze_with_regex function. This approach extracts imports (using require or import syntax), functions, and classes via regular expressions. While less precise than AST parsing, it provides broad cross-language support.
Sources: scripts/analyzers/concept_extractor.py:244-275
Source Text Scanning
A secondary pass, _scan_source_text, searches the entire file content, including comments and string literals, for concept names. Concepts found via this method are assigned a confidence multiplier of 0.7 to account for the higher potential for false positives in non-executable text.
Sources: scripts/analyzers/concept_extractor.py:285-300
Data Models
ExtractedConcept
The primary unit of data produced by the extraction logic.
@dataclass
class ExtractedConcept:
name: str # e.g., "gradient descent"
category: str # algorithm, ml_technique, etc.
confidence: float # 0.0 to 1.0
source_file: str # path to origin
source_line: int # line number (optional)
evidence: str # code snippet triggering detection
Sources: scripts/analyzers/concept_extractor.py:16-25
Mapping and Pattern Recognition
The system relies on predefined mapping dictionaries to translate code signatures into human-readable research concepts.
Import-to-Concept Mapping
The PYTHON_IMPORT_CONCEPTS dictionary maps specific libraries to academic domains. For example, importing transformers triggers "transformer" and "attention mechanism" concepts.
PYTHON_IMPORT_CONCEPTS = {
"transformers": [("transformer", "algorithm", 0.95), ("attention mechanism", "algorithm", 0.9)],
"xgboost": [("gradient boosting", "algorithm", 0.95)],
"gym": [("reinforcement learning", "ml_technique", 0.9)],
"qdrant_client": [("vector database", "technique", 0.8)]
}
Sources: scripts/analyzers/concept_extractor.py:53-108
Code Pattern Recognition
The CODE_PATTERNS list contains regex patterns for common algorithms and data structures. It covers a wide range of computer science topics:
- Sorting/Searching:
quick_sort,binary_search. - Graph Algorithms:
dijkstra,a_star,bfs,dfs. - Machine Learning:
conv2d(CNN),LSTM,GAN,backprop. - Optimization:
simulated_annealing,genetic_algorithm. - Security:
AES,RSA,bcrypt.
Sources: scripts/analyzers/concept_extractor.py:111-146
Integration with MCP Tools
The extraction logic is exposed via the analyze_code tool on the MCP server. When a directory is analyzed, the system aggregates findings across all files using aggregate_concepts.
Conclusion
The Concept Extraction Logic provides a robust mechanism for identifying technical implementation details within source code. By combining deep AST analysis with broad regex patterns and textual scanning, CodeScholar ensures that even obscured research foundations can be identified and linked to relevant academic literature. This system is critical for the project's goal of enriching software development with academic context.
Sources: scripts/generate_knowledge_base.py:820-850, server.js:800-815