Idea Generation Workflow

Relevant source files

The following files were used as context for generating this wiki page:

• main.py
• agents.py
• prompts/deep_research_agent_prompts.py
• searcher/sementic_search.py
• prompts/juder_prompts.py
• LLM.py

The Idea Generation Workflow is a sophisticated, multi-agent process designed to autonomously produce novel research ideas and experimental designs based on existing scientific literature. By leveraging Large Language Models (LLMs) and semantic search capabilities, the system identifies research gaps, analyzes trends, and iteratively refines proposed concepts to ensure they are significant, feasible, and original.

This workflow is primarily managed by the DeepResearchAgent and coordinated through main.py. It utilizes a "Chain-of-Ideas" approach, where research is not treated as a isolated task but as an evolutionary sequence derived from a lineage of related papers.

Sources: main.py:44-53, agents.py:127-142, README.md:12-16

## System Architecture and Components

The workflow relies on several specialized components working in concert to navigate the scientific landscape.

### Core Agents

Component	Description	Primary File
DeepResearchAgent	Responsible for literature search, idea chain construction, and generating the final novel idea.	agents.py
ReviewAgent	Provides critical feedback on generated experiments and identifies potential improvements.	agents.py
SementicSearcher	Handles API interactions with Semantic Scholar to retrieve papers and manage PDF parsing.	searcher/sementic_search.py
LLM Module	Provides the reasoning engine (GPT-4o/mini) for summarization, ideation, and judgment.	LLM.py

Sources: agents.py:53-70, agents.py:101-125, searcher/sementic_search.py:82-95

## The Execution Pipeline

The idea generation process follows a structured sequence from initial topic input to a validated experimental plan.

### Step 1: Literature Search and Chain Construction

The process begins with the generate_idea_with_chain function. It generates search queries based on the provided topic, retrieves relevant papers, and builds a "research chain" by performing both forward search (citations) and backward search (references).

### Step 2: Trend Analysis and Future Direction

Once a chain of papers is established, the agent summarizes the historical evolution (trends) of the research. It then uses specific human-like reasoning patterns—Reflection, Analogy, Deep Dive, and Imitate—to propose future research directions.

Reasoning Mode	Description
Reflection	Identifying significant challenges in current methods and proposing solutions.
Analogy	Applying solutions from other fields to the current research problem.
Deep Dive	Modifying specific aspects of existing methods to enhance effectiveness.
Imitate	Analyzing and evolving the progressive strategies used in previous studies.

Sources: prompts/deep_research_agent_prompts.py:175-195, agents.py:326-340

### Step 3: Idea Generation and Novelty Checking

The agent generates a detailed idea including Motivation, Novelty, and Method. Before finalizing, it performs a strict novelty check against existing literature to avoid plagiarism.

## Experimental Design and Refinement

After an idea is finalized, the system transitions to the experimental phase.

### Experiment Generation

The generate_experiment function takes the idea and related past experiments to design a rigorous validation plan. It includes experimental design, technical implementation details, and feasibility assessments.
Sources: agents.py:448-454, prompts/deep_research_agent_prompts.py:311-332

### Iterative Improvement

The experiment undergoes refinement through a feedback loop between the DeepResearchAgent and the ReviewAgent. The system can even trigger additional literature searches to find specific datasets or metrics mentioned in reviewer suggestions.

## Data Structures

### Result Object

The system uses the Result class to standardize paper information across different modules.
Sources: searcher/sementic_search.py:69-75

Field	Type	Description
title	string	Title of the research paper.
abstract	string	Summary of the paper's content.
article	dict	Full parsed content (sections, references) from scipdf.
citations_count	int	Number of times the paper has been cited.
year	int	Publication year.

## Conclusion

The Idea Generation Workflow represents a closed-loop autonomous system for scientific discovery. By integrating recursive literature searching (agents.py), structured prompt engineering for human-like reasoning (deep_research_agent_prompts.py), and a multi-agent critique system, it minimizes human intervention in the initial stages of research while maintaining a high standard for novelty and feasibility.