Prompt Engineering Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Prompt Engineering Strategy in the CoI-Agent repository defines how Large Language Models (LLMs) are instructed to perform specialized scientific research tasks. The system utilizes a structured multi-agent approach where prompts are modularized into functional categories: research discovery, peer review, and judgmental evaluation. These prompts enforce strict output formats (such as XML-like tags or JSON-formatted strings) to ensure that the outputs can be programmatically parsed and integrated into the research workflow.
The strategy focuses on transforming raw scientific data into structured research components, including novelty assessments, experimental designs, and historical trend analysis. By using specific "Expert" personas (e.g., "scientific research expert," "master of literature searcher"), the prompts guide the LLM to maintain a high level of technical rigor and objective reasoning throughout the "Chain-of-Ideas" generation process.
Research Discovery and Idea Generation Prompts
The core of the system relies on the DeepResearchAgent, which uses a sequence of prompts to move from a general topic to a specific, novel research idea. This process involves searching for literature, extracting information from papers, and analyzing research trends.
Literature Search and Query Refinement
To find relevant papers, the system uses prompts that generate search queries based on topics or existing ideas. If a search fails to yield results, a specific "rewrite" prompt is used to generalize the query and increase the breadth of the search.
| Prompt Function | Purpose | Expected Output Format |
|---|---|---|
get_deep_search_query_prompt | Generates up to 5 diverse literature search queries. | <queries>["q1", "q2"]</queries> |
get_deep_rewrite_query_prompt | Rewrites failed queries to be more concise and general. | <query>{new_query}</query> |
get_deep_check_idea_novel_search_query_prompt | Generates queries specifically to verify the novelty of an idea. | <queries>["q1", "q2"]</queries> |
Sources: prompts/deep_research_agent_prompts.py:6-66
Paper Information Extraction
Once a paper is retrieved, the get_deep_reference_prompt instructs the LLM to act as a scientific expert to extract entities, summarize the background, novelty, and contributions, and detail the experimental process. It also requires the model to identify relevant references within the paper's content.
The prompt explicitly asks for a structured breakdown including "Background," "Novelty," "Contribution," "Detail Reason," and "Limitation."
Sources: prompts/deep_research_agent_prompts.py:69-126, agents.py:206-216
Review and Refinement Prompts
The ReviewAgent utilizes prompts designed to critique proposed ideas and experiments. This ensures the feasibility and originality of the research before finalization.
Experimental Review
The get_review_experiment_design_suggestions_prompt directs the LLM to analyze if an experiment effectively validates its corresponding idea. It evaluates clarity, detail, and potential pitfalls in the methodology.
Iterative Experiment Refinement
When feedback is provided, the system uses refinement prompts to incorporate suggestions. This may involve triggering additional searches for specific technical knowledge to improve the experimental plan.
| Feature | Description | File Reference |
|---|---|---|
| Human Reasoning Modes | Reflection, Analogy, Deep Dive, and Imitate modes for future directions. | deep_research_agent_prompts.py:177-183 |
| Similarity Assessment | Classifying ideas as "plagiarism" if they match existing research fundamentals. | deep_research_agent_prompts.py:317-320 |
| Suggestion Integration | Refining plans based on specific feedback and extracted paper info. | deep_research_agent_prompts.py:377-405 |
Sources: prompts/review_agent_prompts.py:44-71, prompts/deep_research_agent_prompts.py:408-433
Judgment and Evaluation Prompts
The system employs a "Judge" persona to compare different ideas or experiments. These prompts are critical for the ELO-based selection process used in agents.py.
Idea Comparison Criteria
The get_judge_idea_all_prompt provides five distinct criteria for comparing two research ideas:
- Novelty: Uniqueness of problems or approaches.
- Significance: Importance to practitioners/researchers.
- Feasibility: Realization with existing technology.
- Clarity: Organization and writing quality.
- Effectiveness: Likelihood of outperforming baselines.
Sources: prompts/juder_prompts.py:3-43, agents.py:36-50
Implementation and Formatting
All prompts are implemented as Python functions that return formatted strings. These functions often use f-strings to inject context like the topic, idea, or paper_content.
Output Constraints
A recurring strategy across all prompts is the "Strict Format" requirement. Prompts explicitly state: "Please output strictly in the following format (no extra content)." This is vital because the utils.extract function and the agents rely on specific tags (e.g., <think>, <novelty>, <experiment>) to parse responses.
# Example of formatting enforcement in prompts/deep_research_agent_prompts.py:157-160
"""
Please output strictly in the following format:
<trend> {The research trend you summarized based on the past work} </trend>
"""
Sources: prompts/deep_research_agent_prompts.py:157-160, LLM.py:25-36, agents.py:127-133
The Prompt Engineering Strategy effectively bridges the gap between unstructured scientific literature and the structured multi-agent workflow of the CoI-Agent. By defining clear roles, evaluation metrics, and strict output schemas, the project enables autonomous, iterative research development that remains grounded in existing literature while striving for verifiable novelty.