Rubric Configuration & Models

Relevant source files

The following files were used as context for generating this wiki page:

• osmosis_ai/rubric_types.py
• osmosis_ai/rubric_eval.py
• osmosis_ai/cli_services/config.py
• osmosis_ai/cli_services/engine.py
• osmosis_ai/cli_services/session.py
• examples/rubric_configs.yaml
• README.md

The Rubric Configuration & Models system provides a structured framework for defining, parsing, and executing LLM-based evaluations of model outputs. This system allows developers to define natural-language criteria (rubrics) and associate them with specific judge models and scoring parameters to automate reward validation and training workflows.

The system centers around a YAML-based configuration schema that supports versioning, global defaults, and specific model provider integrations via LiteLLM. It ensures strict type enforcement and provides a unified interface for evaluating solutions against reference data.
Sources: README.md:5-10, osmosis_ai/rubric_eval.py:10-15

Data Models and Structures

The system uses several immutable dataclasses to manage the lifecycle of a rubric from configuration file to execution result.

Core Configuration Classes

The primary unit of configuration is the RubricConfig, which encapsulates all necessary data to invoke a judge model for a specific evaluation task.

Class	Description	Key Fields
RubricConfig	Represents a single evaluation task definition.	rubric_id, rubric_text, model_info, score_min, score_max
RubricSuite	A collection of configurations loaded from a single source file.	source_path, version, configs (dict)
ParsedItem	A generic container for raw YAML items before they are validated into configs.	label, payload

Sources: osmosis_ai/cli_services/config.py:22-55

Model Information (ModelInfo)

The ModelInfo type is a dictionary that defines the judge's backend. It requires at least a provider (e.g., "openai", "anthropic") and a model identifier. It also manages authentication via api_key or api_key_env.
Sources: osmosis_ai/rubric_eval.py:465-475, README.md:120-130

Evaluation Results

When an evaluation is executed, the system returns a RewardRubricRunResult.

Rubric Configuration Schema

Rubric configurations are typically stored in YAML files. The system supports versioning to ensure backward compatibility as the schema evolves.

Schema Version 1

Version 1 supports global defaults that apply to all rubrics within the document unless overridden at the individual rubric level.

version: 1
default_score_min: 0.0
default_score_max: 1.0
rubrics:
  - id: support_followup
    rubric: "Evaluate troubleshooting guide effectiveness..."
    model_info:
      provider: openai
      model: gpt-5-mini
      api_key_env: OPENAI_API_KEY

Sources: examples/rubric_configs.yaml:1-15, osmosis_ai/cli_services/config.py:113-130

Configuration Parameters

Parameter	Type	Default	Description
id	string	Required	Unique identifier for the rubric.
rubric	string	Required	Natural language instructions for the judge model.
model_info	mapping	Required	Dictionary containing provider, model, and auth details.
score_min	float	0.0	Minimum possible score.
score_max	float	1.0	Maximum possible score.
system_prompt	string	None	Optional instructions prepended to the judge's system prompt.

Sources: osmosis_ai/cli_services/config.py:145-185, README.md:140-150

Parsing and Resolution Logic

The RubricConfigParser handles the multi-document YAML loading and version-specific schema selection.

Path Discovery

If a configuration path is not explicitly provided to the CLI or SDK, the system searches in the following order:

1. rubric_configs.yaml in the data file's directory.
2. rubric_configs.yaml in the current working directory.
3. examples/rubric_configs.yaml in the current working directory.
   Sources: osmosis_ai/cli_services/config.py:222-235

Evaluation Engine Architecture

The RubricEvaluationEngine coordinates the execution of evaluations across datasets. It uses a RubricEvaluator wrapper to interact with the core evaluate_rubric function.

Execution Flow

1. Context Composition: The engine merges record-level metadata, configuration defaults, and model provider settings into a unified context.
2. API Key Resolution: The system verifies that the required environment variables (e.g., OPENAI_API_KEY) are present.
3. Prompt Construction: The judge's system and user prompts are built using sentinel markers (e.g., <<<BEGIN_CANDIDATE_OUTPUT>>>) to prevent prompt injection from the content being evaluated.
4. Provider Call: LiteLLM invokes the model, enforcing a JSON response format.
5. Sanitization: The response is parsed, <think> blocks are stripped, and the score is clamped to the [score_min, score_max] range.

Sources: osmosis_ai/rubric_eval.py:120-160, osmosis_ai/cli_services/engine.py:40-80, osmosis_ai/rubric_eval.py:240-270

Prompt Structure

The system uses a strict formatting strategy to ensure the judge model treats the model outputs as inert data:

• ORIGINAL_INPUT: The prompt given to the agent.
• CANDIDATE_OUTPUT: The string being evaluated.
• GROUND_TRUTH: Optional reference answer.
• METADATA: Additional structured context.
  Sources: osmosis_ai/rubric_eval.py:170-210, tests/test_rubric_eval.py:5-25

Summary

The Rubric Configuration & Models system provides a robust, type-safe infrastructure for LLM-based evaluation. By centralizing model provider logic via LiteLLM and enforcing a strict YAML-based configuration schema, it enables scalable and reproducible assessment of model performance across varied training and validation tasks.