Extensibility Guide

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/generate_knowledge_base.py
• knowledge/problems/code-quality-assessment.json
• knowledge/problems/code-quality-improvement.json
• knowledge/techniques/capability-maturity-model-integration-cmmi-with-devops-practices.json
• knowledge/papers/devops-integration-with-capability-model-maturity-integration-a-systematic-mappi.json

The CodeScholar system is designed as a multi-pass pipeline for transforming raw research data into a structured 3-layer knowledge architecture. This architecture is specifically engineered for consumption by AI agents via an MCP (Model Context Protocol) wiki. The primary purpose of this guide is to detail how developers can extend the system by adding new research papers, defining novel techniques, and mapping emerging problem domains.

Sources: scripts/generate_knowledge_base.py:10-18

The extensibility of the system relies on a strictly defined JSON schema for each layer: Problems, Techniques, and Papers. By adhering to these schemas, new content can be automatically integrated into the cross-domain connection maps and decision matrices used by the generator script.

Sources: scripts/generate_knowledge_base.py:20-25, knowledge/problems/code-quality-assessment.json

Knowledge Architecture Layers

The system organizes information into three distinct layers. Understanding these layers is critical for extending the repository with new content.

Layer	Description	Key Focus
Layer 1: PROBLEMS	Entry points for users.	Decision matrices and technique rankings.
Layer 2: TECHNIQUES	Implementation details for specific solutions.	Pseudocode, complexity, and parameters.
Layer 3: PAPERS	Academic evidence and benchmarks.	DOI, metrics, and methodology.

Sources: scripts/generate_knowledge_base.py:14-18

Layer 1: Problem Entry Points

Extending the Problems layer involves defining new problem types (e.g., "Code Quality Assessment" or "Workflow Management"). These entries serve as the "decision matrix" for AI agents.

The diagram above shows how a new problem definition maps to multiple techniques based on performance trade-offs.

Sources: knowledge/problems/code-quality-assessment.json:15-45, knowledge/problems/workflow-management-in-software-development.json

Layer 2: Technique Reference Cards

Techniques are synthesized from multiple papers. To extend this layer, developers must provide a canonical name, algorithm steps, and an implementation skeleton.

# Example Implementation Skeleton for a New Technique
def integrate_devops_with_cmmi(literature: List[str]) -> Framework:
    # Step 1: Conduct literature review
    key_practices = identify_key_practices(literature)
    # Step 2: Develop mapping framework
    framework = develop_mapping_framework(key_practices)
    return framework

Sources: knowledge/techniques/capability-maturity-model-integration-cmmi-with-devops-practices.json:52-58

Multi-Pass Pipeline Logic

The generate_knowledge_base.py script utilizes a 4-pass pipeline to process data. Understanding this flow is essential for modifying how the system handles new input.

Pass 1: Extraction Configuration

Pass 1 uses a PAPER_SYSTEM_PROMPT to extract implementable knowledge. To extend the types of data extracted, the JSON schema within this prompt must be updated.

Key Extraction Fields:

• problem_type: Standard CS terminology (e.g., "anomaly detection").
• technique_category: Broad category (e.g., "optimization_algorithm").
• method: Algorithm steps in plain English.
• benchmarks: Datasets, metrics, and baselines.

Sources: scripts/generate_knowledge_base.py:104-160

Pass 4: Cross-Domain Connection Mapping

This pass finds "bridging techniques" that appear in two or more domains. Developers can extend the discovery capabilities by adding new domain tags in the paper metadata.

Sources: scripts/generate_knowledge_base.py:465-485

Integration Guidelines

To add a new research paper to the system:

1. Metadata Registration: Add the paper metadata to unified_library.json (or the equivalent library file).
2. PDF Support: Place the PDF in the specified directory. The system uses PyMuPDF to extract methodology and results sections (up to 30,000 characters).
3. Run Pipeline: Execute the script with specific pass flags to update the knowledge base.

python scripts/generate_knowledge_base.py --pass 1 --skip-existing

Sources: scripts/generate_knowledge_base.py:27-35, scripts/generate_knowledge_base.py:53-70

Conclusion

Extending CodeScholar requires a structured approach to mapping research to implementable software engineering concepts. By contributing new JSON-defined papers, techniques, and problems, developers enhance the AI agent's ability to provide data-driven recommendations and discover emergent cross-domain connections.