Model and Asset Caching
Relevant source files
The following files were used as context for generating this wiki page:
The Model and Asset Caching system is a core infrastructure module in caro. Its primary purpose is to download, store, and manage large assets, particularly LLM models from Hugging Face, to enable offline functionality and improve performance by avoiding repeated downloads. The system is designed to be robust, handling network issues, data integrity, and concurrent access safely.
Beyond models, the caching mechanism is also used for other assets, such as parsed man pages, which are used by the Intelligent Prompt Generation system to validate commands. The cache is managed in a platform-specific, XDG-compliant directory, ensuring predictable behavior across different operating systems.
Sources: specs/003-implement-core-infrastructure/spec.md:13-17, 70-74, specs/006-intelligent-prompt-generation/spec.md:110-113
Architecture and Core Components
The caching system is built around a few key components that work together to provide a seamless caching experience. The main entry point is the CacheManager, which orchestrates all caching operations. It relies on a CacheManifest to keep track of stored assets and uses several data models to represent the state of the cache.
Sources: specs/003-implement-core-infrastructure/tasks.md:15, 100-102, specs/003-implement-core-infrastructure/contracts/cache-api.md
Component Diagram
This diagram shows the relationship between the main components of the caching system.
CacheManager
The CacheManager is the public-facing API for all caching operations. It handles the logic for retrieving, storing, and validating models. It is designed to be thread-safe to handle concurrent access.
Sources: specs/003-implement-core-infrastructure/contracts/cache-api.md:5-29
Public API
| Method | Description | Return Type |
|---|---|---|
new() | Creates a new CacheManager using the default, platform-appropriate cache directory. | Result<Self, CacheError> |
with_cache_dir(PathBuf) | Creates a new CacheManager with a custom cache directory. | Result<Self, CacheError> |
get_model(&str) | Retrieves a model, downloading it from Hugging Face if not present in the cache. | async Result<PathBuf, CacheError> |
is_cached(&str) | Checks if a model is already present in the cache. | bool |
remove_model(&str) | Removes a specific model from the cache. | async Result<(), CacheError> |
clear_cache() | Clears the entire model cache. | async Result<(), CacheError> |
stats() | Retrieves statistics about the current state of the cache. | CacheStats |
validate_integrity() | Validates the integrity of all cached models using checksums. | async Result<IntegrityReport, CacheError> |
Sources: specs/003-implement-core-infrastructure/contracts/cache-api.md:7-29
CacheManifest
The CacheManifest is a persistent data structure, likely a JSON file, that acts as the source of truth for the cache's contents. It tracks all cached models and their metadata, which is crucial for operations like integrity validation, LRU eviction, and providing cache statistics.
Key responsibilities include:
- Adding and removing model entries.
- Retrieving model metadata.
- Performing LRU (Least Recently Used) cleanup when the cache size limit is reached.
- Validating the integrity of the manifest itself.
Sources: specs/003-implement-core-infrastructure/tasks.md:9, 15, specs/003-implement-core-infrastructure/spec.md:104
Data Models
The caching system uses several data structures to represent its state and configuration.
CachedModel
This struct represents a single model stored in the cache, containing all necessary metadata.
| Field | Type | Description |
|---|---|---|
model_id | String | The Hugging Face model identifier (e.g., "Qwen/Qwen2.5-0.5B-Instruct"). |
path | PathBuf | The local filesystem path to the cached model file. |
checksum | String | A 64-character hex string for integrity validation. |
size_bytes | u64 | The size of the model file in bytes. |
downloaded_at | DateTime<Utc> | Timestamp of when the model was downloaded. |
last_accessed | DateTime<Utc> | Timestamp of the last time the model was accessed. |
version | String | The version of the model. |
Sources: specs/003-implement-core-infrastructure/tasks.md:8, specs/003-implement-core-infrastructure/contracts/cache-api.md:41-49
CacheStats and IntegrityReport
These structs provide information about the cache's state and health.
| Struct | Field | Type | Description |
|---|---|---|---|
CacheStats | total_models | usize | The total number of models in the cache. |
total_size_bytes | u64 | The total disk space used by the cache. | |
cache_dir | PathBuf | The path to the cache directory. | |
models | Vec<CachedModelInfo> | A list of all cached models with their metadata. | |
IntegrityReport | valid_models | Vec<String> | A list of model IDs that passed integrity checks. |
corrupted_models | Vec<String> | A list of model IDs that failed integrity checks. | |
missing_models | Vec<String> | A list of model IDs present in the manifest but not on disk. |
Sources: specs/003-implement-core-infrastructure/contracts/cache-api.md:32-58
CacheConfig
The cache behavior can be configured via the main config.toml file.
| Field | Type | Description |
|---|---|---|
cache_dir | PathBuf | The root directory for all cached assets. |
man_page_ttl_days | u32 | Time-to-live for cached man pages before they are refreshed. |
auto_refresh | bool | If true, the cache will automatically refresh expired assets. |
Sources: specs/006-intelligent-prompt-generation/spec.md:468-474
Caching Logic and Flows
get_model Workflow
The get_model function is the core of the caching logic. It ensures a model is available locally, downloading it only when necessary.
Sequence Diagram: First-Time Model Request
This diagram illustrates the sequence of events when a user's command requires a model that is not yet cached.
Configuration
The cache directory can be configured through multiple methods, providing flexibility for users and CI/CD environments.
- CLI Flag: The
--cache-diroption can be used to specify a custom cache directory for a single command execution. - Environment Variable: The
HF_HOMEenvironment variable is respected, which aligns with the broader Hugging Face ecosystem. - Default Location: If not otherwise specified,
carouses a platform-specific default directory, such as~/.cache/huggingface/hub, following the XDG Base Directory Specification on Linux/macOS.
Sources: exports/session-logs/2025-09-13-readme.txt:46, 131-136, specs/003-implement-core-infrastructure/spec.md:105
Error Handling
The CacheError enum defines a set of typed errors to clearly communicate failures during cache operations. This allows the application to handle failures gracefully, for example by retrying a download or notifying the user with an actionable message.
| Error Variant | Description |
|---|---|
DirectoryNotAccessible | The cache directory could not be read from or written to due to permission issues. |
ModelNotFound | The requested model does not exist on the remote Hugging Face Hub. |
DownloadFailed | A network error occurred during the download, or the system is offline. |
ChecksumMismatch | The downloaded file is corrupt as its checksum does not match the expected value. |
DiskFull | There is not enough disk space to download or store the model. |
Io | A generic I/O error occurred during file operations. |
Serialization | An error occurred while serializing or deserializing the cache manifest. |
Sources: specs/003-implement-core-infrastructure/contracts/cache-api.md:61-88
Other Cached Assets
Man Page Cache
The caching infrastructure is not limited to LLM models. It is also used to store structured data parsed from system man pages. This cache is used by the ValidationAgent to confirm that flags and options in a generated command are valid for the user's specific platform (e.g., BSD ls vs. GNU ls).
The ManPageCache is stored in a JSON file within the cache directory and has its own TTL (Time-To-Live) configuration to control how often it's refreshed.
Sources: specs/006-intelligent-prompt-generation/spec.md:110-113, 120-136