Juder (Judge) Agent
Relevant source files
The following files were used as context for generating this wiki page:
The Juder (Judge) Agent is a critical evaluative component within the Chain-of-Ideas (CoI) Agent framework. Its primary purpose is to provide objective, multi-dimensional assessments of research ideas and experimental designs generated by other agents in the system. By utilizing specific prompts and Large Language Models (LLMs), it acts as an arbiter to determine the quality, novelty, and feasibility of various research outputs.
The agent operates as a functional module that compares competing ideas to select the most promising direction for further development. This selection process is often integrated into the DeepResearchAgent workflow to filter and rank a pool of generated ideas using scoring mechanisms like Elo.
Sources: agents.py:44-55, prompts/juder_prompts.py:2-42
Architecture and Core Logic
The Juder Agent's logic is primarily encapsulated in the judge_idea asynchronous function and the prompt templates defined in juder_prompts.py. It does not exist as a standalone class but as a specialized service used by the DeepResearchAgent.
Evaluation Flow
The evaluation process follows a structured path where two ideas (or experiments) are compared side-by-side based on a predefined research topic. The LLM processes these through a "thinking" phase before outputting structured scores.
The diagram above shows the typical flow of an evaluation task handled by the Judge logic.
Sources: agents.py:44-55, prompts/juder_prompts.py:35-41
Key Evaluation Dimensions
The Juder Agent evaluates ideas based on five specific technical criteria:
| Dimension | Description |
|---|---|
| Novelty | Assessment of whether the problem or approach is new or a novel combination of techniques. |
| Significance | The importance of the idea and its potential for use by other researchers or practitioners. |
| Feasibility | Whether the idea can be realized with existing technology and is logically sound. |
| Clarity | Evaluation of how well-organized and clearly written the proposal is. |
| Effectiveness | The likelihood that the proposed idea will perform better than existing baselines. |
Sources: prompts/juder_prompts.py:12-16, agents.py:48-53
Technical Implementation
Idea Comparison Function
The judge_idea function is the primary entry point for idea evaluation. It utilizes the openai_llm to process prompts and the extract utility to parse structured XML-like tags from the response.
async def judge_idea(i, j, idea0, idea1, topic, llm):
prompt = get_judge_idea_all_prompt(idea0, idea1, topic)
messages = [{"role":"user", "content":prompt}]
response = await llm.response_async(messages)
# Extraction of scores from LLM response
novelty = extract(response, "novelty")
relevance = extract(response, "relevance")
significance = extract(response, "significance")
# ... further extraction
return i, j, novelty, relevance, significance, clarity, feasibility, effectiveness
Sources: agents.py:44-55, LLM.py:145-163
Scoring and Ranking Mechanism
When multiple ideas are generated, the DeepResearchAgent conducts a round-robin tournament where every idea is compared against every other idea using the judge. The winners are determined by converting the judge's categorical "choice" into numerical scores.
- Choice 0: Idea 0 wins (+1 point).
- Choice 1: Idea 1 wins (+1 point).
- Choice 2: Equal/Tie (+0.5 point each).
Sources: agents.py:168-185
Experiment Evaluation
In addition to ideas, the Juder system provides prompts for evaluating experimental designs through get_judge_experiment_all_prompt. This ensures that the implementation plan for a research idea is as robust as the idea itself.
Experiment Criteria
The judge evaluates experiments based on:
- Feasibility: Can the experiment be realized with existing methods?
- Quality: Is there a clear rationale for each step and appropriate baseline selection?
- Clarity: Does the plan provide enough information for an expert reader to understand the steps?
Sources: prompts/juder_prompts.py:44-77
Prompt Engineering Guidelines
The Juder Agent's prompts include strict instructions to mitigate bias:
- Position Bias: Judges are instructed to ignore the order of presentation.
- Verbosity Bias: Judges are cautioned not to allow the length of a response to influence the evaluation.
- Format Enforcement: Output must strictly follow the
<tag>value</tag>structure for automated parsing.
Sources: prompts/juder_prompts.py:19-21, prompts/juder_prompts.py:35-41
Summary
The Juder (Judge) Agent provides the critical "selective pressure" in the Chain-of-Ideas framework. By systematically evaluating ideas and experiments across novelty, significance, and feasibility dimensions, it ensures that only high-quality, scientifically sound research directions are proposed as final outputs. It bridges the gap between raw LLM generation and expert-level research standards through rigorous comparison and scoring.
Sources: agents.py:190-195, main.py:45-55