Data Ingestion Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Data Ingestion Pipeline is a multi-stage system designed to transform raw academic paper metadata and unstructured content into a structured, 3-layer knowledge architecture. Its primary purpose is to provide AI agents with implementable technical knowledge categorized into Problems, Techniques, and Papers.
The pipeline manages the transition from external sources like Journal Club or Semantic Scholar into a unified library, followed by deep extraction of technical insights using LLMs (Large Language Models) and PDF analysis.
Sources: scripts/generate_knowledge_base.py:1-15, scripts/import_existing_papers.py:1-10
Pipeline Architecture
The ingestion process is structured as a multi-pass workflow that handles data normalization, content extraction, and high-level synthesis.
Stage 1: Initial Import and Normalization
Raw data from external sources is converted into a unified CodeScholar format. This stage focuses on a "Metadata + Abstract" approach to minimize storage requirements (saving approximately 1.2 GB) while maintaining links to full papers via DOIs.
The diagram above shows the linear progression from raw data import to the four distinct processing passes of the knowledge base generator.
Sources: scripts/generate_knowledge_base.py:17-26, scripts/import_existing_papers.py:115-140
Stage 2: Knowledge Extraction (Pass 1)
Pass 1 utilizes LLMs (specifically GPT-4o) to extract "implementable knowledge." If available, the pipeline uses PyMuPDF (fitz) to extract text from PDF files, focusing on methodology, results, and algorithm sections.
Key components of this pass include:
- PDF Text Extraction: Retrieves the first 30,000 characters from up to 15 pages.
- Structured Schema: Outputs JSON containing
algorithm_steps,key_parameters,complexity, andimplementation_guide.
Sources: scripts/generate_knowledge_base.py:64-90, scripts/generate_knowledge_base.py:130-185
Stage 3: Synthesis and Clustering (Pass 2 & 3)
The pipeline aggregates information across multiple papers to create higher-level abstractions:
- Technique Cards: Synthesized from multiple papers describing the same algorithm. Includes implementation skeletons and common mistakes.
- Problem Entry Points: Created by clustering techniques under standard CS problem types (e.g., "object detection", "code generation").
Sources: scripts/generate_knowledge_base.py:20-24, scripts/generate_knowledge_base.py:330-360
Data Models and Formats
Unified Paper Schema
The pipeline enforces a strict schema for all ingested papers to ensure compatibility with downstream search and analysis tools.
| Field | Type | Description |
|---|---|---|
id | UUID | Unique identifier generated for the paper. |
title | String | Original paper title. |
abstract | String | Paper description or abstract used for semantic search. |
identifiers | Object | Contains doi, doi_url, and semantic_scholar_id. |
metadata | Object | Stores published_date, year, and citation_count. |
files | Object | Tracks pdf_url and local path status. |
Sources: scripts/import_existing_papers.py:61-105
Extraction JSON Schema
The extraction process targets specific technical fields required for software engineering applications.
{
"technique_name": "Canonical name",
"method": {
"approach": "High-level description",
"algorithm_steps": ["Step 1", "Step 2"],
"key_parameters": ["param: value"]
},
"implementation_guide": {
"data_structures": ["List of structures"],
"pseudocode_hint": "Python-like syntax"
}
}
Sources: scripts/generate_knowledge_base.py:134-180, knowledge/papers/evaluation-of-prompt-engineering-on-the-performance-of-a-large-language-model-in.json:20-50
Search and Retrieval Integration
The data ingested by the pipeline is indexed by the server.js component for real-time querying.
- Hybrid Search: Combines keyword matching with vector embeddings generated from paper abstracts.
- Vectra Index: A local vector index is used for semantic search if embeddings are present.
- Concept Extraction: Analyzes source code to identify technical concepts that match the ingested "Problem Types" and "Techniques."
Sources: server.js:45-75, server.js:108-150
This sequence illustrates how data flows from the ingestion scripts into the active server environment for agent consumption.
Sources: server.js:47-65, scripts/import_existing_papers.py:155-165
Component Summary
| Component | Responsibility | Source File |
|---|---|---|
import_existing_papers.py | Normalizes legacy Journal Club metadata. | scripts/import_existing_papers.py |
generate_knowledge_base.py | Orchestrates LLM extraction and synthesis passes. | scripts/generate_knowledge_base.py |
extract_pdf_text | Parses local PDF files for context enrichment. | scripts/generate_knowledge_base.py:64 |
paper_to_markdown | Generates human-readable wiki pages for papers. | scripts/generate_knowledge_base.py:220 |
concept_extractor.py | Extract concepts from code (referenced in server). | server.js:525 |
Sources: scripts/generate_knowledge_base.py, scripts/import_existing_papers.py, server.js
The Data Ingestion Pipeline ensures that research knowledge is not just archived but transformed into an actionable format for developers and AI agents, specifically focusing on implementation details like complexity, parameters, and pseudocode.