Model and Adapter Management
Relevant source files
The following files were used as context for generating this wiki page:
The Model and Adapter Management system in monGARS provides a centralized mechanism for configuring, provisioning, and accessing Large Language Models (LLMs) and their associated adapters. It is designed to be flexible, supporting multiple model providers (like Ollama and local paths) and allowing for different model configurations through a profile system. This ensures that the application can consistently access the correct models for various tasks, such as general conversation or coding, while abstracting away the complexities of model installation and management.
The system is primarily orchestrated by the LLMModelManager class, which reads configuration from a JSON file, handles model and adapter downloads, and provides an interface for other parts of the application to retrieve model information. It is exposed to operators via a REST API and a command-line script for easy inspection and provisioning.
Architecture Overview
The management system is composed of several key classes that work together to handle model and adapter lifecycles.
LLMModelManager: The central orchestrator. It loads model profiles from a configuration file, applies environment variable overrides, and manages the provisioning process.AdapterProvisioner: A helper class used byLLMModelManagerspecifically for handling the download, installation, and verification of adapter artifacts (e.g., LoRA weights).ModelSlotManager: Works in conjunction withLLMModelManagerto coordinate adapter lifecycles and update worker pools, particularly in distributed environments like Ray.- Configuration File (
llm_models.json): A JSON file that defines model profiles, including the models for different roles, their providers, and any associated adapters.
The following diagram illustrates the relationships between these core components.
This diagram shows how the LLMModelManager acts as a central hub, consuming configuration and using the AdapterProvisioner to manage artifacts, while being controlled by the CLI and API.
Sources: monGARS/core/model_manager.py, monGARS/core/adapter_provisioner.py, scripts/generate_monGARS_llm_dataset.py:381-399
Configuration
Model and adapter configurations are defined in a JSON file, typically configs/llm_models.json. This file contains profiles, which are collections of model definitions for different roles.
Configuration Schema
The configuration is structured around three main data classes: ModelProfile, ModelDefinition, and AdapterDefinition.
This class diagram shows that a ModelProfile contains multiple ModelDefinition objects, and each ModelDefinition can have multiple AdapterDefinition objects.
Sources: monGARS/core/model_manager.py:28-100
ModelDefinition
This class describes a single logical model role.
| Attribute | Type | Description |
|---|---|---|
role | str | The logical role of the model (e.g., "general", "coding"). |
name | str | The name of the model, specific to the provider. |
provider | str | The model provider, e.g., "ollama" or "local_path". Default is "ollama". |
parameters | Mapping | A dictionary of provider-specific parameters. |
auto_download | bool | If True, the model will be downloaded automatically if missing. Default is True. |
description | str | An optional description of the model. |
adapters | Tuple[AdapterDefinition] | A tuple of adapter definitions associated with this model. |
Sources: monGARS/core/model_manager.py:58-72
AdapterDefinition
This class describes a LoRA or other adapter artifact.
| Attribute | Type | Description |
|---|---|---|
name | str | The name of the adapter. |
source | str | The source URL or path to download the adapter from. |
target | str | The destination path for the adapter within the registry. |
checksum | str | An optional checksum to verify the adapter artifact. |
auto_update | bool | If True, the adapter will be updated automatically. Default is True. |
description | str | An optional description of the adapter. |
Sources: monGARS/core/model_manager.py:28-41
Environment Variable Overrides
The active profile and specific model names can be overridden using environment variables:
LLM_MODELS_CONFIG_PATH: Path to the JSON model configuration file.LLM_MODELS_PROFILE: Name of the profile to use.LLM_MODELS_AUTO_DOWNLOAD: Global flag to enable/disable automatic downloads.LLM_GENERAL_MODEL: Overrides the name of the model for the "general" role.LLM_CODING_MODEL: Overrides the name of the model for the "coding" role.
Sources: README.md:169-178, monGARS/core/model_manager.py:190-193, 622-634
Provisioning Workflow
Models and adapters are provisioned via the ensure_models_installed method in LLMModelManager. This can be triggered from the CLI or the API. The process involves checking for the existence of the required models/adapters and downloading/installing them if they are missing and auto-download is enabled.
The following sequence diagram illustrates the flow for a provisioning request made through the API.
This diagram shows the sequence of operations, from the user's API request to the LLMModelManager coordinating with providers and the AdapterProvisioner to ensure all artifacts are available.
Sources: monGARS/core/model_manager.py:255-283, monGARS/core/adapter_provisioner.py, tests/test_api_model_management.py:109-121
Provisioning Report
The result of a provisioning action is a ModelProvisionReport, which contains a list of ModelProvisionStatus objects.
ModelProvisionStatus Attribute | Type | Description |
|---|---|---|
role | str | The role being provisioned (e.g., general or general:my-adapter). |
name | str | The name of the model or adapter. |
provider | str | The provider (ollama, local_path, adapter). |
action | str | The action taken (e.g., exists, installed, updated, skipped, error). |
detail | str | Additional details, such as auto_download_disabled or an error message. |
Sources: monGARS/core/model_manager.py:103-129
API Endpoints
The model management system is exposed via a set of admin-only REST API endpoints.
| Method | Endpoint | Description |
|---|---|---|
GET | /api/v1/models | Returns the active model profile, its model definitions, and a list of all available profiles. Requires admin privileges. |
POST | /api/v1/models/provision | Triggers the provisioning process for specified model roles. Requires admin privileges. |
POST /api/v1/models/provision Request Body:
{
"roles": ["general", "coding"],
"force": true
}
roles(Optional): A list of roles to provision. If omitted, all roles in the active profile are provisioned.force(Optional): A boolean to force re-downloading/re-installing even if the model exists.
Sources: README.md:182-188, tests/test_api_model_management.py:87-121
Command-Line Interface (CLI)
A CLI script is provided to trigger the provisioning workflow from the shell.
Usage:
python -m scripts.provision_models --roles general coding
The script uses LLMModelManager to perform the same provisioning logic as the API.
| Argument | Description |
|---|---|
--roles | A space-separated list of model roles to provision. If not provided, all roles are provisioned. |
--profile | Specifies which configuration profile to use. |
--force | Forces re-downloading of models even if they already exist. |
--json | Emits a machine-readable JSON report of the provisioning actions. |
Sources: README.md:187-188, scripts/provision_models.py
Summary
The Model and Adapter Management system is a critical component for maintaining a flexible and robust AI application. By decoupling model configuration from the core application logic and providing clear interfaces for management via API and CLI, monGARS allows operators to easily swap models, update adapters, and manage different deployment environments without code changes. The use of profiles, provider abstractions, and automated provisioning ensures that the system is both powerful and easy to use.