Knowledge Base Schema

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/generate_knowledge_base.py
• knowledge/README.md
• knowledge/papers/a-hereditary-attentive-template-based-approach-for-complex-knowledge-base-questi.json
• knowledge/techniques/knowledge-based-planning.json
• knowledge/problems/complex-knowledge-base-question-answering.json

The CodeScholar Knowledge Base is a structured, multi-layer research architecture designed specifically for consumption by AI agents via the Model Context Protocol (MCP). It transforms raw academic paper metadata and PDF content into a hierarchy of implementable technical knowledge, allowing software engineers to move from a problem statement to a specific, evidence-backed algorithm.

The system organizes information into three primary layers: Problems (decision entry points), Techniques (algorithmic reference cards), and Papers (empirical evidence and benchmarks). These layers are interconnected to facilitate emergent discovery across different research domains.

Sources: scripts/generate_knowledge_base.py:16-24, knowledge/README.md:3-8

3-Layer Architecture Overview

The knowledge base follows a top-down structure designed to support different engineering workflows, from high-level problem identification to low-level implementation details.

Layer 1: Problem Schema

The Problems layer serves as the entry point for users. It defines a "Problem Entry Point" document that helps engineers determine if their current technical challenge matches a known problem type and provides a decision matrix for selecting an appropriate technique.

Problem Data Model

Field	Type	Description
problem_name	String	Canonical name for the problem type.
description	String	Plain English explanation of the problem scope.
you_have_this_if	List[String]	Indicators/symptoms of the problem.
approaches	List[Object]	Summary of techniques applied to this problem, including paper counts and tradeoffs.
decision_matrix	List[Object]	Comparison of techniques based on Speed, Memory, Accuracy, and Ease of Implementation.
start_here	String	Recommended first-choice approach with reasoning.

Sources: scripts/generate_knowledge_base.py:532-563, knowledge/problems/complex-knowledge-base-question-answering.json:2-38

Layer 2: Technique Schema

The Techniques layer consists of "Reference Cards" that synthesize knowledge from multiple papers into a definitive guide for a specific algorithm or framework. This layer prioritizes implementation details like pseudocode and complexity over experimental results.

Technique Data Model

Field	Type	Description
technique_name	String	The canonical name (e.g., "NSGA-II", "BERT").
one_liner	String	High-level summary of what the technique does.
how_it_works	String	Detailed explanation for a senior engineer.
algorithm	Object	Contains steps (pseudocode) and core_equation.
parameters	List[Object]	Tunable parameters with typical values and their effects.
complexity	Object	Big-O time and space complexity and practical notes.
implementation_skeleton	String	Python-like pseudocode showing core operations.
maturity	String	Status: established, proven, emerging, or experimental.

Sources: scripts/generate_knowledge_base.py:339-376, knowledge/techniques/knowledge-based-planning.json:2-101

Layer 3: Paper Schema

The Papers layer contains structured extractions from individual academic works. It provides the empirical foundation for the higher layers, focusing on specific benchmarks, datasets, and implementation "gotchas" identified by the original authors.

Knowledge Extraction Pipeline

The knowledge base is generated through a four-pass pipeline executed by scripts/generate_knowledge_base.py. This pipeline uses LLM-based extraction (specifically GPT-4o) to transform unstructured text into the schemas defined above.

Pipeline Stages

1. Pass 1: Deep Paper Extraction: Processes individual papers (via abstracts or full PDF text) to create structured JSON files.
2. Pass 2: Cluster & Synthesize: Groups papers by technique_name to generate unified Technique Reference Cards.
3. Pass 3: Problem Mapping: Clusters techniques by problem_type to create Problem Entry Points and Decision Matrices.
4. Pass 4: Cross-Domain Mapping: Identifies "bridging techniques" that appear across multiple domains (e.g., an optimization algorithm used in both Robotics and NLP).

Sources: scripts/generate_knowledge_base.py:26-31, scripts/generate_knowledge_base.py:612-788

Conclusion

The Knowledge Base Schema creates a formal structure for academic knowledge that favors engineering utility over bibliometric data. By enforcing strict schemas for problems, techniques, and papers, CodeScholar enables AI agents to provide context-aware recommendations that are grounded in peer-reviewed research and practical implementation constraints.

Sources: knowledge/README.md:10-20, scripts/generate_knowledge_base.py:821-825