LLM-as-a-Judge Configuration
Relevant source files
The following files were used as context for generating this wiki page:
LLM-as-a-Judge Configuration defines the parameters and logic used to evaluate model outputs using a secondary Large Language Model (LLM). This system automates the grading process by providing the "judge" model with specific criteria, scoring rubrics, and context to assess the quality of responses generated by a "candidate" model.
The configuration ensures consistency across evaluations by standardizing how the judge interprets prompts and assigns scores. It manages the interaction between the evaluation platform, the judge model provider (such as OpenAI or Anthropic), and the specific metrics being measured.
Judge Configuration Architecture
The configuration architecture relies on a structured schema that defines the judge's behavior, including the prompt template and the scoring mechanism.
The diagram shows the hierarchical relationship between the configuration and the components that drive the evaluation logic.
Sources: eval_platform/core/judge_config.py, eval_platform/schemas/judge.py
Key Components
The system uses several data structures to manage judge settings:
- JudgeModel: Defines the specific LLM instance used as the grader, including its version and provider-specific parameters (e.g., temperature).
- EvaluationPrompt: Contains the system and user instructions that guide the judge's assessment.
- ScoringSchema: Outlines the numerical or categorical scale used to quantify the candidate model's performance.
Sources: eval_platform/models/judge.py:10-45, eval_platform/schemas/judge.py:5-20
Evaluation Workflow
When an evaluation starts, the platform initializes the judge configuration to process candidate responses. The judge receives the original input, the candidate's output, and the reference ground truth (if available).
The sequence diagram illustrates the flow of data from the platform through the configuration to the judge model.
Sources: eval_platform/api/v1/endpoints/judges.py:50-85
Configuration Parameters
The following table summarizes the primary configuration options available for LLM judges:
| Parameter | Type | Description |
|---|---|---|
model_name | String | The identifier for the LLM used as the judge (e.g., gpt-4o). |
temperature | Float | Controls the randomness of the judge's response; typically set low (0.0). |
prompt_template | String | The skeleton text containing placeholders for inputs and outputs. |
max_tokens | Integer | Limits the length of the judge's reasoning and final score output. |
retry_limit | Integer | The number of times to re-attempt evaluation if the judge fails. |
Sources: eval_platform/core/judge_config.py:15-30, eval_platform/schemas/judge.py:22-40
Implementation Details
The core logic for parsing judge responses resides in the JudgeConfig class. It extracts the score and reasoning from the LLM's raw text output using regex or JSON parsing.
class JudgeConfig(BaseModel):
name: str
provider: str
model: str
template: str
def format_prompt(self, input_text: str, output_text: str) -> str:
return self.template.format(input=input_text, output=output_text)
Sources: eval_platform/core/judge_config.py:5-12
API Endpoints
The system exposes REST endpoints to manage judge configurations:
POST /judges/: Creates a new judge configuration with specific templates.GET /judges/{judge_id}: Retrieves details for a specific grader.PUT /judges/{judge_id}: Updates prompts or model parameters.
Sources: eval_platform/api/v1/endpoints/judges.py:10-35
Summary
LLM-as-a-Judge Configuration provides the necessary infrastructure to automate complex evaluations. By standardizing prompts and scoring rubrics through a centralized configuration system, the platform ensures that model assessments are reproducible and scalable across different evaluation datasets.