Custom & Local Model Setup
Relevant source files
The following files were used as context for generating this wiki page:
The Custom & Local Model Setup module enables the evaluation platform to interface with Large Language Models (LLMs) hosted outside of standard cloud providers. This system integrates local inference engines and private API endpoints into the evaluation workflow.
You use this module to benchmark models running on local hardware or within private VPCs. The setup handles model registration, connection strings, and authentication for non-standard model providers.
Sources: README.md, app/services/model_loader.py
Infrastructure and Deployment
The platform uses Docker containers to manage local model environments. The orchestration layer defines how local inference services communicate with the evaluation core.
The diagram shows the routing logic between the evaluation core and different model hosting environments.
Local Service Configuration
The system defines local service dependencies through a containerized architecture. This ensures that the evaluation environment has access to the necessary drivers and local network ports.
| Parameter | Type | Description |
|---|---|---|
LOCAL_MODEL_URL | String | The endpoint for the locally hosted inference engine (e.g., Ollama or vLLM). |
MODEL_PATH | String | The filesystem path to the model weights for direct loading. |
GPU_ENABLED | Boolean | Flag to trigger CUDA/ROCm support in the container. |
Sources: docker-compose.yaml:12-25, app/core/config.py:45-50
Model Registration Workflow
To use a custom model, you must register its metadata in the system. The platform validates the connection before allowing evaluation tasks to proceed.
This sequence illustrates the validation process during custom model registration.
Custom Provider Implementation
The model_loader.py service manages the initialization of different model classes. It checks the provider field in the model configuration to determine which driver to instantiate.
- Define the model provider in the
ModelRequestobject. - Provide the base URL for the custom API.
- Supply required headers or API keys if the custom endpoint requires authentication.
Sources: app/services/model_loader.py:10-35, app/api/models.py:5-15
Environment Variables for Local Setup
Configuration for local models relies on environment variables defined in the core settings. These variables control timeout durations and retry logic for local endpoints which may have variable latency.
| Variable Name | Default Value | Purpose |
|---|---|---|
CUSTOM_MODEL_TIMEOUT | 60 | Maximum seconds to wait for a custom model response. |
LOCAL_RETRY_COUNT | 3 | Number of attempts to reconnect to a local service. |
ALLOW_UNVERIFIED_SSL | False | Permits connections to custom endpoints with self-signed certificates. |
Sources: app/core/config.py:102-115
Summary
Custom and local model setup allows the platform to extend evaluation capabilities beyond standard cloud APIs. By configuring local endpoints and registering custom model providers, you can benchmark private models while maintaining data sovereignty and reducing latency for high-volume evaluation tasks.