Introduction to Eval-Platform

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• docker-compose.yml
• backend/app/main.py
• backend/app/models/eval.py
• backend/app/api/v1/api.py
• frontend/package.json

Eval-Platform provides a standardized environment for evaluating Large Language Model (LLM) performance. The system enables developers to run benchmarks, track evaluation metrics, and manage datasets through a centralized interface.

The platform coordinates interactions between a web-based frontend, a FastAPI backend, and various LLM providers. It stores evaluation results in a relational database to facilitate historical analysis and comparison of model versions.

Sources: README.md:1, backend/app/main.py

System Architecture

The platform uses a microservices-oriented architecture orchestrated via Docker. This structure separates the user interface, the API logic, and the persistent data storage.

Core Components

Component	Technology	Responsibility
Frontend	React / Next.js	Provides the user interface for managing datasets and viewing results.
Backend	FastAPI (Python)	Processes evaluation logic and communicates with LLM APIs.
Database	PostgreSQL	Stores metadata, user configurations, and evaluation records.
Cache/Queue	Redis	Manages asynchronous evaluation tasks and session state.

Sources: docker-compose.yml:5-45, frontend/package.json:5-15

High-Level Data Flow

The following diagram illustrates the flow of a typical evaluation request from the user interface to the backend and external LLM providers.

The diagram shows the asynchronous nature of the evaluation process where the API acknowledges the request immediately while the worker handles external LLM calls.

Sources: docker-compose.yml:10-30, backend/app/api/v1/api.py

Data Models

The system organizes information around Evaluations, Datasets, and Models. These entities are represented as SQLAlchemy models in the backend.

Evaluation Entity

An Evaluation record tracks the performance of a specific model against a dataset. It contains configuration parameters and the final calculated metrics.

Field	Type	Description
id	UUID	Primary key for the evaluation record.
status	String	Current state (PENDING, RUNNING, COMPLETED, FAILED).
model_id	String	Identifier for the LLM being tested.
dataset_id	UUID	Foreign key to the source dataset.
metrics	JSONB	Aggregate scores like accuracy, latency, or custom metrics.

Sources: backend/app/models/eval.py:10-40

API Structure

The backend exposes a RESTful API organized by version. Most interactions occur through the /api/v1 prefix.

Key Endpoints

1. POST /evaluations: Initiates a new evaluation task.
2. GET /evaluations/{id}: Retrieves the status and results of a specific evaluation.
3. GET /datasets: Lists available test datasets.
4. POST /datasets/upload: Uploads new JSON or CSV files for benchmarking.

Sources: backend/app/api/v1/api.py:5-25

Deployment Configuration

The platform uses Docker Compose for local development and production deployment. The configuration defines networking and environment variables required for service communication.

version: '3.8'
services:
  db:
    image: postgres:15-alphine
    environment:
      - POSTGRES_DB=eval_db
  backend:
    build: ./backend
    depends_on:
      - db
      - redis
    env_file: .env

Sources: docker-compose.yml:1-25

Summary

Eval-Platform centralizes LLM evaluation by providing a robust API and UI for benchmarking. It utilizes a FastAPI backend to handle complex evaluation logic and a PostgreSQL database for persistent result tracking. Through its containerized architecture, it ensures consistent environments for both development and production use cases.