Evaluation Engine Design
Relevant source files
The following files were used as context for generating this wiki page:
The Evaluation Engine manages the lifecycle of automated assessments within the platform. It executes benchmarks against target models, processes scoring logic, and aggregates results for analysis. The engine operates as a backend service that interfaces with model providers and data storage to ensure consistent and reproducible evaluations.
This system coordinates the transition of evaluations through various states, from initial queuing to final reporting. It abstracts the complexity of different evaluation metrics and model communication protocols, providing a unified interface for users to trigger and monitor performance tests.
Architecture and Components
The engine follows a modular architecture consisting of API controllers, business logic services, and core execution modules. This separation ensures that request handling, state management, and the actual evaluation logic remain decoupled.
The diagram above illustrates the request flow from the API layer down to the execution engine and data persistence.
Sources: backend/app/api/endpoints/evaluations.py, backend/app/services/eval_service.py, backend/app/core/engine.py
Key Components
| Component | Responsibility |
|---|---|
| Evaluations API | Handles HTTP requests, validates input schemas, and returns task status. |
| Eval Service | Manages database transactions, orchestrates high-level workflow, and handles errors. |
| Core Eval Engine | Executes the actual model prompts and calculates scores using defined metrics. |
| Evaluation Model | Defines the data structure for storing evaluation metadata and results. |
Sources: backend/app/models/evaluation.py, backend/app/schemas/evaluation.py, backend/app/services/eval_service.py
Execution Workflow
When a user initiates an evaluation, the system follows a structured sequence to ensure reliability. The EvalService creates a record in the database before passing the task to the CoreEngine for asynchronous processing.
This sequence diagram depicts the asynchronous nature of the evaluation execution, where the user receives an immediate response while the engine processes the task.
Sources: backend/app/api/endpoints/evaluations.py:10-45, backend/app/services/eval_service.py:25-60
Data Model and Schemas
The system uses specific Pydantic schemas for input validation and SQLAlchemy models for persistence. The Evaluation model tracks the model version, dataset used, and the final score.
Evaluation Attributes
- id: Unique identifier for the evaluation task.
- model_id: The target model being evaluated.
- dataset_id: The reference dataset used for testing.
- status: Current state (e.g.,
PENDING,RUNNING,COMPLETED,FAILED). - metrics: A JSON field storing various performance indicators (Accuracy, F1, Latency).
Sources: backend/app/models/evaluation.py:5-30, backend/app/schemas/evaluation.py:1-25
Logic and Scoring
The CoreEngine iterates through the dataset, sending inputs to the specified model and comparing the outputs against ground truth.
- Retrieval: Fetch the next batch of test cases from the dataset.
- Inference: Send the prompt to the Model Provider API.
- Scoring: Apply metric functions (e.g., exact match, semantic similarity) to the model response.
- Aggregation: Calculate the mean or weighted average across all test cases.
Sources: backend/app/core/engine.py:40-85
Summary
The Evaluation Engine provides a robust framework for quantifying model performance. By decoupling the API, service, and core execution logic, the platform supports scalable and maintainable evaluation workflows. The use of asynchronous processing ensures that long-running benchmarks do not block the user interface, while the structured data model enables detailed historical tracking and comparison of model versions.