Dataset Management

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• backend/app/models/dataset.py
• backend/app/api/v1/endpoints/datasets.py
• backend/app/schemas/dataset.py
• backend/app/crud/crud_dataset.py

Dataset Management handles the storage, organization, and retrieval of evaluation data within the platform. It provides the infrastructure to upload raw data, convert it into structured formats, and manage metadata required for evaluation runs.

The system supports versioning and categorization to ensure that different evaluation experiments use consistent and reproducible data inputs.

Data Model

The core of the management system resides in the Dataset database model. This model defines the attributes and relationships necessary for tracking data assets.

Dataset Attributes

Field	Type	Description
id	Integer	Unique identifier for the dataset
name	String	Human-readable name of the dataset
description	String	Optional details regarding the dataset content
file_path	String	Storage location of the raw data file
created_at	DateTime	Timestamp of initial upload
updated_at	DateTime	Timestamp of the last modification

Sources: backend/app/models/dataset.py

Architecture and Flow

Dataset Management follows a standard CRUD (Create, Read, Update, Delete) pattern integrated with the backend API and database layer.

Upload and Storage Flow

The following diagram illustrates how the system processes a new dataset from the user interface to the persistent storage.

The system validates incoming data against Pydantic schemas before the CRUD controller persists metadata to the database and saves the physical file to the configured storage path.

Sources: backend/app/api/v1/endpoints/datasets.py, backend/app/schemas/dataset.py

API Endpoints

The platform exposes RESTful endpoints to interact with datasets. These endpoints allow users to list available data, retrieve specific details, or add new evaluation sets.

Endpoint Specifications

Method	Path	Description
GET	/datasets/	Retrieves a list of all available datasets
POST	/datasets/	Uploads and registers a new dataset
GET	/datasets/{id}	Fetches detailed metadata for a specific dataset
DELETE	/datasets/{id}	Removes a dataset and its associated files

Sources: backend/app/api/v1/endpoints/datasets.py

Implementation Details

Data Validation

The system uses Pydantic models to enforce data integrity during API requests. This ensures that names are strings and that required fields are present before processing.

class DatasetCreate(DatasetBase):
    name: str
    description: Optional[str] = None
    file_path: str

Sources: backend/app/schemas/dataset.py:10-13

CRUD Operations

The CRUD layer abstracts database interactions, providing a clean interface for the API layer to perform operations like create, get_multi, and remove.

Summary

Dataset Management provides a centralized system for handling evaluation inputs. By combining structured metadata in a relational database with flexible file storage, the platform ensures that evaluation runs are based on verified and organized data assets.