Model Management
Relevant source files
The following files were used as context for generating this wiki page:
Model Management handles the lifecycle and configuration of machine learning models within the evaluation platform. It provides the necessary infrastructure to register, update, and track different model versions and their associated metadata.
This system serves as the central registry for all Large Language Models (LLMs) and custom models that the platform evaluates. Users interact with this module to define model parameters, provider details (such as OpenAI or Anthropic), and specific endpoint configurations.
Architecture and Data Flow
Model Management follows a standard CRUD (Create, Read, Update, Delete) pattern integrated with a FastAPI backend and a PostgreSQL database. The data flow begins at the API layer, moves through schema validation, and is persisted via the CRUD layer using SQLAlchemy models.
The diagram above illustrates the request-response cycle for managing model entities.
Sources: backend/app/api/v1/endpoints/models.py, backend/app/crud/crud_model.py
Data Structures
The system uses specific Pydantic schemas for data validation and SQLAlchemy models for database representation.
Model Schema Fields
The following table describes the primary fields used in model definitions:
| Field | Type | Description |
|---|---|---|
id | Integer | Unique identifier for the model |
name | String | Display name of the model |
provider | String | The model provider (e.g., 'openai', 'huggingface') |
model_path | String | Path or identifier used to call the model |
description | String | Detailed information about the model's purpose |
config | JSON | Model-specific parameters like temperature or top_p |
Sources: backend/app/schemas/model.py:5-25, backend/app/models/model.py:10-20
Database Entity Relationship
The Model entity maintains relationships with evaluation tasks and results to track performance over time.
This relationship allows the platform to aggregate metrics per model across different evaluation suites.
Sources: backend/app/models/model.py
API Endpoints
The Model Management API provides the primary interface for external systems and frontend components to interact with model data.
Endpoint Overview
GET /models/: Retrieves a list of all registered models with pagination support.POST /models/: Registers a new model in the system.GET /models/{model_id}: Fetches detailed information for a specific model.PUT /models/{model_id}: Updates configuration or metadata for an existing model.DELETE /models/{model_id}: Removes a model from the registry.
Sources: backend/app/api/v1/endpoints/models.py:15-60
Registration Logic
When you register a new model, the system validates the input against the ModelCreate schema before the CRUD layer executes the database insertion.
CRUD Implementation
The CRUDModel class extends a base CRUD class to provide specific logic for model entities. It ensures that model identifiers are unique and handles the serialization of configuration JSON objects.
# Example of model creation logic
def create(self, db: Session, *, obj_in: ModelCreate) -> Model:
db_obj = Model(
name=obj_in.name,
provider=obj_in.provider,
model_path=obj_in.model_path,
config=obj_in.config,
description=obj_in.description
)
db.add(db_obj)
db.commit()
db.refresh(db_obj)
return db_obj
Sources: backend/app/crud/crud_model.py:10-22
Summary
Model Management provides the core registry for all LLMs used in the evaluation platform. By centralizing model configurations and providers through a structured API and database schema, it ensures consistent evaluation environments. The system facilitates the tracking of model metadata and supports the integration of various external model providers.