Terminology & Core Concepts
Relevant source files
The following files were used as context for generating this wiki page:
The eval-platform manages the assessment of Large Language Model (LLM) outputs through structured evaluation workflows. This system coordinates the interaction between datasets, models, and evaluators to produce quantitative and qualitative metrics.
The platform provides a centralized environment where users upload datasets, configure evaluation parameters, and execute automated or manual scoring. This ensures consistency across different model versions and prompt iterations.
Evaluation Entity Relationships
The core of the platform relies on the relationship between Datasets, Models, and the resulting Evaluation Reports.
The diagram shows how a user initiates an Evaluation Job using a specific Dataset and Model to produce Results and Metrics.
Sources: apps/server/src/database/schema.prisma, apps/web/src/types/index.ts
Core Terminology
Dataset
A Dataset represents a collection of input prompts or queries used to test a model. Each entry typically contains a unique identifier, the prompt text, and optionally a reference answer (ground truth).
Sources: apps/web/src/types/index.ts:10-25, apps/server/src/database/schema.prisma:45
Model Configuration
Model Configuration defines the specific LLM being tested and its hyper-parameters. This includes the provider (e.g., OpenAI, Anthropic), model version, temperature, and max tokens.
Sources: apps/web/src/types/index.ts:30-45, apps/server/src/database/schema.prisma:60
Evaluation Job
An Evaluation Job is a single execution instance where a Model processes a Dataset. It tracks the lifecycle of the evaluation from "pending" to "completed" or "failed".
Sources: apps/server/src/modules/evaluation/evaluation.service.ts:100-120, apps/web/src/lib/api.ts:50
Evaluation Workflow
The evaluation process follows a strict sequence to ensure data integrity and model performance tracking.
The flowchart illustrates the data flow from job creation through model inference to final report generation.
Sources: apps/server/src/modules/evaluation/evaluation.service.ts:150-180
System Components Summary
| Component | Responsibility | Relevant Data |
|---|---|---|
| Ingestion Engine | Validates and stores uploaded datasets. | CSV/JSON formats, row counts. |
| Inference Wrapper | Communicates with LLM providers to get responses. | API keys, model parameters. |
| Scoring Module | Applies logic (exact match, LLM-as-a-judge) to responses. | Scores, feedback strings. |
| Analytics Dashboard | Visualizes performance across different runs. | Latency, cost, accuracy. |
Sources: README.md, apps/web/src/lib/api.ts
Metric Definitions
The platform calculates various metrics to quantify model quality.
- Accuracy: The percentage of model outputs that match the reference answer exactly or within a specific threshold.
- Latency: The time taken (in milliseconds) for the model to generate a complete response.
- Cost: The estimated financial expenditure based on token usage and provider pricing.
- Custom Scorers: User-defined logic or secondary LLM prompts used to grade subjective attributes like "helpfulness" or "tone".
Sources: apps/server/src/modules/evaluation/evaluation.service.ts:200-225, apps/web/src/types/index.ts:80-95
Execution States
Jobs transition through specific states during their lifecycle.
| State | Description |
|---|---|
PENDING | The job is queued and waiting for available workers. |
RUNNING | The model is currently processing dataset rows. |
COMPLETED | All rows are processed and metrics are calculated. |
FAILED | An error occurred during inference or scoring. |
Sources: apps/server/src/database/schema.prisma:15-20, apps/web/src/types/index.ts:50
This terminology and conceptual framework provide the foundation for consistent evaluation and comparison of LLM outputs within the platform.