Evaluation Metrics & Scoring Data

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/api/metrics.py
• src/models/scoring.py
• src/services/eval_engine.py
• src/schemas/evaluation.py
• src/core/config.py

The Evaluation Metrics & Scoring Data module manages the calculation, storage, and retrieval of performance indicators for AI models. This system provides a standardized framework for quantifying model accuracy, latency, and reliability across different datasets.

By centralizing scoring logic, the platform ensures consistent benchmarking. The module interacts with the evaluation engine to process raw model outputs and transform them into structured metric records.

Sources: src/services/eval_engine.py:10-25, src/models/scoring.py:5-12

Metric Architecture

The platform uses a modular architecture to define and execute metrics. Each metric is encapsulated as a processor that takes model predictions and ground truth labels as input.

Metric Types and Components

Component	Type	Description
AccuracyMetric	Statistical	Measures the percentage of correct predictions.
LatencyMetric	Performance	Records the time taken for inference in milliseconds.
F1ScoreMetric	Statistical	Calculates the harmonic mean of precision and recall.
CustomMetric	Extensible	Allows users to define domain-specific scoring logic via scripts.

Sources: src/api/metrics.py:45-60, src/schemas/evaluation.py:20-35

Scoring Data Flow

The scoring data flow follows a synchronous pipeline where predictions are validated before metric calculation begins.

This diagram illustrates the progression from raw model output to the final visualization in the dashboard.

Sources: src/services/eval_engine.py:100-115

Data Models

The system relies on structured data models to maintain the integrity of scoring history. The primary entities include ScoreRecord and MetricDefinition.

ScoreRecord Schema

The ScoreRecord stores individual evaluation results linked to a specific model version and dataset.

Field	Type	Constraint	Description
id	UUID	Primary Key	Unique identifier for the score.
eval_id	UUID	Foreign Key	Reference to the evaluation run.
metric_name	String	Not Null	The name of the metric (e.g., "accuracy").
value	Float	Check (>=0)	The numerical result of the calculation.
timestamp	DateTime	Default Now	When the score was generated.

Sources: src/models/scoring.py:20-45

Metric Relationships

The following entity relationship diagram defines how scores relate to evaluation runs and models.

The ER diagram shows that one evaluation run can produce multiple score records, each tied to a specific metric definition.

Sources: src/models/scoring.py:50-70

Implementation Logic

The evaluation engine coordinates the scoring process by iterating through the registered metrics for a given project configuration.

Metric Calculation Sequence

When an evaluation job starts, the engine follows this sequence:

1. Retrieve evaluation configuration from the database.
2. Load the specified metric processors.
3. Fetch the model prediction batch.
4. Execute each processor against the batch.
5. Aggregate results into a summary report.

The sequence diagram highlights the loop used to process each prediction against the selected metrics.

Sources: src/services/eval_engine.py:150-185, src/core/config.py:30-45

Code Example: Metric Registration

The following snippet demonstrates how the system registers a new metric calculation.

# src/api/metrics.py:75-82
def register_metric(name: str, func: Callable):
    """Registers a new metric function to the global registry."""
    if name in METRIC_REGISTRY:
        raise ValueError(f"Metric {name} already exists")
    METRIC_REGISTRY[name] = func

Sources: src/api/metrics.py:75-82

Summary

The Evaluation Metrics & Scoring Data system provides the backbone for model assessment within the platform. It ensures that every model prediction is processed through a validated pipeline, resulting in immutable score records. Through its modular architecture, the system supports both standard statistical metrics and custom performance indicators, allowing developers to track model evolution over time accurately.