Experiment Tracking

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• backend/app/models/experiment.py
• backend/app/api/v1/endpoints/experiments.py
• backend/app/schemas/experiment.py
• backend/app/crud/crud_experiment.py

Experiment Tracking manages the lifecycle of evaluation runs within the platform. It records configuration parameters, execution states, and performance metrics generated during the assessment of language models.

This system provides a centralized repository for comparing different model versions, prompt templates, and dataset configurations. It ensures reproducibility by capturing the exact environment and inputs used for every evaluation instance.

Data Model and Storage

The core of experiment tracking relies on a structured data model that captures the relationship between models, datasets, and results.

Experiment Entity

The Experiment model defines the attributes stored for each evaluation run.

Field	Type	Description
id	UUID	Unique identifier for the experiment
name	String	Human-readable name assigned to the run
status	Enum	Current state (PENDING, RUNNING, COMPLETED, FAILED)
model_id	UUID	Reference to the model being evaluated
dataset_id	UUID	Reference to the dataset used for testing
params	JSON	Configuration parameters (temperature, top_p, etc.)
metrics	JSON	Aggregated performance scores
created_at	DateTime	Timestamp of experiment initialization

Sources: backend/app/models/experiment.py:1-25, backend/app/schemas/experiment.py:5-20

Entity Relationships

The following diagram illustrates how experiments connect to models and datasets.

The diagram shows that an experiment acts as a junction point between a specific model version and a dataset.
Sources: backend/app/models/experiment.py:10-18

Experiment Lifecycle

Experiments progress through a series of states from initiation to completion. The CRUD layer handles these transitions while the API layer exposes controls to the user.

State Transitions

1. Initialization: A user submits a configuration via the POST /experiments/ endpoint. The system creates a record with PENDING status.
2. Execution: The task runner picks up the experiment, updating the status to RUNNING.
3. Finalization: Upon completion, the runner updates the metrics field and sets the status to COMPLETED. If an error occurs, it sets the status to FAILED.

API Interface

The system exposes RESTful endpoints to manage and retrieve experiment data.

Method	Endpoint	Description
GET	/experiments/	List all experiments with pagination and filtering
POST	/experiments/	Create a new experiment run
GET	/experiments/{id}	Retrieve detailed configuration and metrics for a specific run
DELETE	/experiments/{id}	Remove an experiment record and its associated results

Sources: backend/app/api/v1/endpoints/experiments.py:10-80

Configuration Example

When creating an experiment, the platform requires a JSON payload defining the target model and parameters.

{
  "name": "GPT-4 Baseline Evaluation",
  "model_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "dataset_id": "7bc95f64-2217-4562-b3fc-2c963f66afb2",
  "params": {
    "temperature": 0.7,
    "max_tokens": 512
  }
}

Sources: backend/app/schemas/experiment.py:22-35

Summary

Experiment Tracking enables systematic evaluation by recording every interaction between models and datasets. By persisting configuration parameters and performance metrics, it allows developers to identify optimal model settings and track improvements over time. The module integrates with the CRUD layer for database persistence and provides a standardized API for external interaction.