Review Agent
Relevant source files
The following files were used as context for generating this wiki page:
The Review Agent is a specialized module within the Chain-of-Ideas (CoI) framework designed to evaluate, critique, and provide constructive feedback on generated research ideas and experimental designs. Functioning as an automated manuscript reviewer, it leverages Large Language Models (LLMs) to assess the feasibility, novelty, and scientific relevance of research proposals against existing literature. Sources: agents.py:58-64, prompts/review_agent_prompts.py:19-25
The scope of the Review Agent includes searching for related literature to verify originality, analyzing the soundness of experimental steps, and providing iterative suggestions that the Deep Research Agent uses to refine its outputs. It serves as a critical quality control gate in the autonomous research workflow. Sources: agents.py:108-112, main.py:48-52
Architecture and Components
The Review Agent is implemented as a Python class ReviewAgent that integrates with semantic search tools and LLM providers. It maintains internal state for logging suggestions and managing paper context. Sources: agents.py:58-76
Key Components
| Component | Description | Source |
|---|---|---|
SementicSearcher | Interface used to retrieve and read academic papers from Semantic Scholar. | agents.py:72, searcher/sementic_search.py:64 |
openai_llm | The primary LLM engine (e.g., GPT-4o) used for complex reasoning and review generation. | agents.py:12-25, LLM.py:46 |
log_save_file | Directory where review logs and experiment suggestions are persisted in JSON format. | agents.py:61-64 |
System Flow
The following diagram illustrates how the Review Agent interacts with the overall system to refine a research experiment.
The diagram shows the iterative feedback loop between the Deep Research Agent and the Review Agent. Sources: agents.py:431-447, main.py:51-54
Core Logic and Evaluation Functions
The Review Agent utilizes specific prompting strategies to simulate expert peer review. Its logic is divided into literature-based idea assessment and structural experiment validation.
Literature-Based Review
The agent generates search queries based on the target idea to find potential "prior art." It then compares the idea's novelty against the retrieved abstracts and titles. Sources: agents.py:86-93, prompts/review_agent_prompts.py:1-17
- Function:
get_search_query(idea, topic) - Prompt:
get_review_search_related_paper_prompt - Logic: Outputs up to 3 queries to check for existing explorations of the idea. Sources: agents.py:86-93, prompts/review_agent_prompts.py:4-16
Experiment Validation
This process focuses on the internal consistency and feasibility of the proposed methodology. It evaluates whether the experiment can effectively verify the hypothesis. Sources: agents.py:108-116, prompts/review_agent_prompts.py:43-62
The sequence shows the asynchronous call to the LLM and the subsequent logging of the review results. Sources: agents.py:108-116
Evaluation Criteria
When reviewing experiments, the agent applies specific technical criteria:
- Validation Capability: Can the experiment actually prove the core idea?
- Procedure Clarity: Are steps confusing or poorly designed?
- Feasibility: Is the dataset construction and method implementation realistic?
- Constructiveness: Suggestions must be specific without altering the original idea's intent. Sources: prompts/review_agent_prompts.py:51-60
Integration with Idea Selection
The Review Agent logic is also utilized during the idea selection phase, where multiple ideas are judged against each other using a set of comparative metrics. Sources: agents.py:34-45, prompts/juder_prompts.py:1-30
Multi-Dimensional Scoring
Ideas are judged based on the following dimensions:
- Novelty: Are the approaches new or a novel combination?
- Significance: Is the work important to other researchers?
- Feasibility: Can it be realized with existing technology?
- Clarity: Is the proposal well-organized?
- Effectiveness: Likelihood of outperforming baselines. Sources: prompts/juder_prompts.py:10-14, agents.py:43-45
Implementation Details
Class Structure
The ReviewAgent class manages the state of the review process, including the history of suggestions provided during a single run. Sources: agents.py:58-76
class ReviewAgent:
def __init__(self, save_file="saves/", llm=None, cheap_llm=None, publicationData=None, **kwargs) -> None:
self.paper_save_file = os.path.join(save_file, "review_papers")
self.log_save_file = os.path.join(save_file, "review_logs")
self.llm = llm
self.cheap_llm = cheap_llm
self.reader = SementicSearcher(self.paper_save_file)
self.review_experiment_suggestions = [] # Stores history
Sources: agents.py:58-76
Async Workflow
The agent is designed to run within an asyncio loop, allowing it to perform concurrent LLM requests and paper retrievals. Sources: agents.py:86, main.py:48-52
Summary
The Review Agent acts as the "critical eye" of the CoI-Agent system. By combining semantic literature search with specialized LLM prompts for peer review, it ensures that generated research ideas are not only novel but also supported by logically sound and feasible experimental designs. It provides the necessary feedback loop that allows the system to improve its outputs through iterative refinement. Sources: agents.py:431-447, prompts/review_agent_prompts.py:43-62