Model Catalog and Loading
Relevant source files
The following files were used as context for generating this wiki page:
The Model Catalog and Loading system in caro is responsible for the discovery, acquisition, caching, and loading of Large Language Models (LLMs) used for command generation. It is designed to provide a "batteries-included" experience by shipping with a default embedded model, ensuring zero-config, offline-first functionality. The system also supports optional remote backends like Ollama and vLLM, with a robust fallback mechanism to the embedded model.
This architecture ensures that caro is immediately usable after installation without requiring network access or external service configuration, while still offering power users the flexibility to leverage more powerful, remote models. The core components manage everything from downloading models from Hugging Face Hub to validating their integrity and loading them into the appropriate inference runtime.
Architecture Overview
The system's architecture prioritizes the embedded model as the default and fallback, with remote backends acting as optional enhancements. The BackendSelector initiates the process, which then involves the ModelLoader and CacheManager to ensure the correct model is available for the selected inference backend.
This diagram shows the high-level data flow for model selection and loading.
Core Components
The model loading system is composed of several key modules and data structures that work together.
ModelLoader
The ModelLoader is the central component responsible for the model distribution strategy. It handles both locating the bundled embedded model and downloading remote models if they are not present in the local cache.
Key responsibilities include:
- Platform Detection: Determines the appropriate model variant (
MLXorCPU) to use based on the user's hardware. - Model Downloading: Integrates with the
hf-hubcrate to download models from Hugging Face Hub, complete with progress bars. - Path Management: Provides paths to the correct model files, whether they are bundled or cached.
Key methods defined for ModelLoader:
get_embedded_model_path() -> Result<PathBuf>: Returns the path to the bundled model.download_model_if_missing(variant: ModelVariant) -> Result<PathBuf>: Downloads a model from Hugging Face Hub if it's not already in the cache.detect_platform() -> ModelVariant: ReturnsMLXfor Apple Silicon aarch64 orCPUfor other platforms.
Sources: specs/004-implement-ollama-and/tasks.md:100-112
CacheManager
The CacheManager handles all interactions with the local model cache, which is located at ~/.cache/caro/models/. It is designed to support offline operations by ensuring that once a model is downloaded, it can be loaded without network access.
Functional requirements for the cache module include:
- FR-C001: Download models and store them in a platform-specific cache directory (respecting XDG specifications).
- FR-C002: Detect if a model is already cached to avoid re-downloading.
- FR-C003: Support offline operation by loading models directly from the cache.
- FR-C004: Provide a mechanism for users to clear the cache.
- FR-C005: Validate the integrity of cached models using checksums before loading.
The implementation uses tokio::fs for asynchronous file I/O and sha2 for checksum validation.
Sources: specs/003-implement-core-infrastructure/spec.md:94-98, specs/003-implement-core-infrastructure/tasks.md:173-182
Model Types
caro supports two primary types of models: a default embedded model for immediate use and optional remote models for enhanced performance or quality.
Embedded Model
The embedded model ensures caro works out-of-the-box with zero configuration.
- Default Model: The primary embedded model is Qwen2.5-Coder-1.5B, chosen for its balance of speed and shell command accuracy. Other models like Phi-3 and StarCoder2 are used for benchmarking.
- Inference Variants: The system supports two runtimes for the embedded model:
- MLX: For Apple Silicon (M1/M2/M3/M4) devices, providing GPU-accelerated performance. This is the primary build for macOS.
- CPU (Candle/Burn): A cross-platform CPU-based inference runtime that serves as a fallback for non-Apple hardware (Linux, Windows).
- Offline Capability: The embedded model is designed to work entirely offline.
Sources: specs/004-implement-ollama-and/plan.md:43-57, 217-220, specs/004-implement-ollama-and/spec.md:149-152
Remote Backends
Users can optionally configure caro to use remote backends like Ollama or vLLM. The model loading system is designed to gracefully fall back to the embedded model if a configured remote backend is unavailable. This logic is handled by the BackendSelector.
Sources: specs/004-implement-ollama-and/plan.md:21-26, specs/004-implement-ollama-and/spec.md:41-44
Data Models
Several data structures are central to the model catalog and loading process.
This class diagram illustrates the key data models and their relationships.
| Struct/Enum | Description | Source File |
|---|---|---|
CachedModel | Represents a locally stored model file with metadata for integrity checking. | specs/003-implement-core-infrastructure/spec.md:135 |
CacheManifest | Tracks all models in the cache, their metadata, and total size. Manages LRU eviction. | specs/003-implement-core-infrastructure/spec.md:139 |
ModelVariant | Enum to differentiate between MLX (Apple Silicon GPU) and CPU (cross-platform) inference backends. | specs/004-implement-ollama-and/tasks.md:178-189 |
EmbeddedConfig | Configuration for the embedded model, including temperature, max tokens, and stop tokens. | specs/004-implement-ollama-and/tasks.md:204-218 |
ModelCache | A data model representing the model cache, likely a precursor to CacheManifest. | specs/001-create-a-comprehensive/tasks.md:83 |
Model Loading Sequence
The following diagram details the step-by-step process from a user running a command to a model being loaded and ready for inference.
Configuration
Users can override the default model loading behavior via the ~/.config/caro/config.toml file. This allows specifying a different default backend or providing a custom path for the embedded model.
| Configuration Key | Type | Description | Default |
|---|---|---|---|
default_backend | string | The default backend to use. Can be "embedded", "ollama", or "vllm". | "embedded" |
embedded.variant | string | The inference variant for the embedded model. Can be "MLX" or "CPU". | Auto-detected |
embedded.model_path | path | An optional override path to the embedded model file. | Bundled path |
cache_max_size_gb | integer | The maximum size of the model cache in gigabytes. | 10 |
Sources: specs/004-implement-ollama-and/tasks.md:434-451, specs/003-implement-core-infrastructure/tasks.md:121-125
Summary
The Model Catalog and Loading system is a cornerstone of caro's design, enabling a seamless user experience that is both powerful and accessible. By providing a high-performance embedded model for Apple Silicon and a reliable CPU fallback, it guarantees offline functionality out-of-the-box. The system's ability to integrate with remote backends and manage a local cache provides the flexibility and performance that power users expect, making caro a versatile tool for a wide range of developers.
Sources: mlx-test/DELIVERABLES.md, specs/004-implement-ollama-and/plan.md