Custom Evaluation Metrics Plugin

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• backend/app/api/v1/endpoints/metrics.py
• backend/app/core/plugins/base.py
• backend/app/services/evaluator.py
• backend/app/schemas/metric.py
• backend/app/models/metric.py

The Custom Evaluation Metrics Plugin system provides a modular framework for extending the evaluation capabilities of the platform. It allows developers to define, register, and execute specialized scoring logic beyond the built-in defaults. The system ensures that new metrics integrate seamlessly with the evaluation pipeline while maintaining consistent data interfaces for inputs and results.

This plugin architecture isolates evaluation logic from the core platform, enabling independent development and testing of new metrics. The platform utilizes these plugins to process model outputs and generate quantitative scores based on user-defined criteria.

Sources: README.md, backend/app/core/plugins/base.py

Plugin Architecture and Base Interface

The system defines a rigid contract for custom metrics through the BaseMetricPlugin class. Every plugin must inherit from this base class and implement its abstract methods to ensure compatibility with the EvaluatorService.

Core Components

• Plugin Registry: Manages the discovery and instantiation of available metric plugins.
• BaseMetricPlugin: An abstract base class that defines the required interface, including metadata properties and the execution method.
• Metric Schema: Defines the standard JSON structure for passing data to plugins and receiving calculated scores.

Sources: backend/app/core/plugins/base.py:10-45, backend/app/schemas/metric.py:5-20

Plugin Execution Flow

The following diagram illustrates how the EvaluatorService interacts with the plugin system during a measurement task.

The diagram shows the lifecycle of a metric execution from discovery to result validation.
Sources: backend/app/services/evaluator.py:50-85, backend/app/core/plugins/base.py

Implementation Requirements

To create a new metric, you must provide specific metadata and implement the scoring logic. The platform uses this metadata for display in the UI and for selecting appropriate metrics during job configuration.

Required Methods and Attributes

Attribute/Method	Type	Description
name	String	Unique identifier for the metric.
version	String	Semantic versioning string for the plugin.
description	String	Human-readable explanation of the metric's logic.
score(input, output, target)	Method	Processes data and returns a MetricResult object.

Sources: backend/app/core/plugins/base.py:15-30, backend/app/models/metric.py:12-18

Sequence of Metric Calculation

The sequence diagram below details the interaction between the API, the Evaluation Service, and a specific Plugin instance.

This diagram describes the synchronous scoring loop used by the backend to process evaluation batches.
Sources: backend/app/api/v1/endpoints/metrics.py:25-50, backend/app/services/evaluator.py:100-125

Data Models and Schemas

The plugin system relies on Pydantic models to enforce type safety during the exchange of information between the core system and external plugins.

Metric Configuration

Plugins receive a configuration object that includes parameters specific to their logic, such as thresholds or weights.

class MetricConfig(BaseModel):
    name: str
    params: Dict[str, Any] = {}
    threshold: Optional[float] = None

Sources: backend/app/schemas/metric.py:10-15

Metric Result

The output of every plugin must conform to the MetricResult schema, which includes the raw score and optional diagnostic metadata.

class MetricResult(BaseModel):
    score: float
    reasoning: Optional[str] = None
    latency: float
    metadata: Dict[str, Any] = {}

Sources: backend/app/schemas/metric.py:22-28, backend/app/models/metric.py:40-45

Registration and Integration

The platform automatically discovers plugins located in the designated plugins/ directory. Each plugin must be registered within the __init__.py of the module to be visible to the PluginManager.

1. Place the plugin file in backend/app/core/plugins/custom/.
2. Define the class inheriting from BaseMetricPlugin.
3. Register the class in the local registry.
4. Restart the backend service to finalize discovery.

Sources: backend/app/core/plugins/base.py:50-60, README.md

The Custom Evaluation Metrics Plugin system ensures the platform remains extensible and adaptable to various LLM evaluation needs. By standardizing the interface through BaseMetricPlugin, the system allows for robust, type-safe integration of diverse scoring methodologies.