Evaluation Datasets

Relevant source files

The following files were used as context for generating this wiki page:

• dataset/case.json
• agents.py
• README.md
• prompts/deep_research_agent_prompts.py
• searcher/sementic_search.py
• prompts/juder_prompts.py

Evaluation Datasets in the Chain-of-Ideas (CoI) Agent project serve as the foundational benchmark for assessing the ability of Large Language Model (LLM) agents to generate novel, feasible, and scientifically relevant research ideas. These datasets consist of curated scientific literature, entity-centric knowledge stores, and structured "cases" that represent the historical progression and future potential of specific research topics.

The primary purpose of these datasets is to facilitate a "Chain-of-Ideas" reasoning process, where agents analyze past work to identify research trends, limitations, and future directions. This data is utilized by the DeepResearchAgent and ReviewAgent to validate generated hypotheses against existing benchmarks such as MOSES or expert-generated ideas.

Sources: README.md:46-47, dataset/case.json:1-124, agents.py:108-112

Dataset Structure and Components

The evaluation data is primarily organized within the dataset/ directory. A central component is the structured JSON case file, which encapsulates the entire lifecycle of a research idea from motivation to experimental validation.

Case Data Model

Each case in the dataset is structured to provide a comprehensive context for evaluation. This includes the generated idea, the chain of literature it was derived from, and the specific entities involved.

Field	Type	Description
title	String	The title of the generated research idea.
idea	String	Detailed breakdown including Origins, Motivation, Novelty, and Methodology.
idea_chain	String	A numbered sequence of existing papers and their summarized contributions.
related_experiments	List	A collection of experimental steps derived from the papers in the idea_chain.
entities	String	Key technical terms, models, and datasets (e.g., GCT, SMILES, VAE).
trend	String	Textual analysis of the progression between papers in the chain.
future	String	Proposed future research directions based on the analyzed trend.

Sources: dataset/case.json:1-124

Data Flow for Evaluation

The following diagram illustrates how evaluation datasets are processed by the system to generate and validate new research ideas.

The system ingest case data to build an idea chain, analyzes trends, and iteratively refines the output via a reviewing agent.
Sources: agents.py:166-235, dataset/case.json:115-124

Source Material and Acquisition

Evaluation datasets are built using high-impact scientific literature sourced from external APIs and parsed into internal structures.

Semantic Search and PDF Parsing

The project utilizes the Semantic Scholar Academic Graph API to retrieve papers published within specific timeframes. These papers are then parsed using scipdf_parser to extract structured sections (Abstract, Introduction, Results).

• Key Class: SementicSearcher in searcher/sementic_search.py manages API calls and PDF downloads.
• Search Fields: The system specifically retrieves title, abstract, citationCount, and openAccessPdf to ensure evaluation quality.

Sources: searcher/sementic_search.py:73-85, README.md:16-24

Entity-Centric Knowledge Store

A critical part of the evaluation dataset is the extraction of "Entities." These include model names (e.g., GPT-4), datasets (e.g., MOSES), and specialized terminology (e.g., SMILES strings). These entities provide the "ground truth" context for the LLM to ground its generated ideas.

Sources: dataset/case.json:101-114, prompts/deep_research_agent_prompts.py:89-94

Evaluation Metrics and Criteria

The datasets are used in conjunction with a JudgeAgent (via juder_prompts.py) to score generated ideas against the benchmark data.

Scoring Dimensions

The evaluation process uses five specific metrics to compare generated ideas against the "gold standard" in the dataset:

1. Novelty: Assessment of whether the approach is a new combination or addresses a new problem.
2. Significance: Potential for others to build upon the idea or its impact on the field.
3. Feasibility: Logical consistency and availability of technology to realize the idea.
4. Clarity: Quality of organization and communication of the research plan.
5. Effectiveness: Likelihood of the idea outperforming existing baselines found in the dataset.

Sources: prompts/juder_prompts.py:11-18, agents.py:46-52

Experimental Validation Benchmarks

The dataset includes related_experiments which serve as procedural baselines. For example, in molecular generation tasks, the dataset references the MOSES platform for evaluating validity, uniqueness, and similarity.

Sources: dataset/case.json:94-97, prompts/deep_research_agent_prompts.py:246-255

Summary of Case Components

Component	File Reference	Function
Idea Chain	dataset/case.json	Maps the evolution of a topic (e.g., Paper 0 to Paper 4).
Search Queries	deep_research_agent_prompts.py	Templates used to find relevant papers for the dataset.
Review Suggestions	review_agent_prompts.py	Feedback loops used to refine the dataset cases.
Result Metadata	searcher/sementic_search.py	Stores titles, abstracts, and citation counts for ranking.

Sources: dataset/case.json:76-114, searcher/sementic_search.py:65-71, prompts/review_agent_prompts.py:27-35

Evaluation Datasets in this repository are not static collections but structured frameworks that enable the LLM agent to simulate the human research process of literature review, gap identification, and experimental design. By organizing data into idea_chains and entities, the system provides a benchmark for autonomous scientific discovery.