Caching Strategies
Relevant source files
The following files were used as context for generating this wiki page:
LiteLLM provides a robust caching system designed to optimize performance and reduce costs by storing and reusing responses from Large Language Model (LLM) API calls. The system supports multiple backends, ranging from local in-memory storage to distributed solutions like Redis and S3, ensuring flexibility for various deployment environments.
The caching module intercepts requests to generate a unique cache key based on request parameters. If a hit occurs, the cached result is returned immediately, bypassing the LLM provider. This system is integrated into both synchronous and asynchronous workflows across completion, embedding, and transcription endpoints.
Architecture and Components
The caching architecture is centered around a unified Cache class that acts as a manager for specific backend implementations. It leverages a standard interface (BaseCache) to interact with different storage providers.
Core Cache Management
The Cache class initializes the selected backend and handles high-level logic such as cache key generation, TTL management, and deciding whether a request should be cached based on the global CacheMode.
Sources: litellm/caching/caching.py:46-177, litellm/caching/caching.py:649-657
Key Components
| Component | Description |
|---|---|
Cache | Main manager class for all caching operations. |
InMemoryCache | Local, non-persistent cache stored in the process memory. |
RedisCache | Distributed cache backend using Redis or Redis Cluster. |
S3Cache | Persistent object-based cache using AWS S3. |
DualCache | Hybrid strategy updating both Redis and InMemoryCache simultaneously. |
RedisSemanticCache | Advanced cache using vector similarity for partial matches. |
Sources: litellm/caching/caching.py:100-146, litellm/caching/Readme.md:1-12
Data Flow for Cache Operations
The following diagram illustrates the flow when a completion request is made with caching enabled.
The system generates a SHA-256 hash of parameters like model, messages, and temperature to create the key.
Sources: litellm/caching/caching.py:186-213, litellm/caching/caching.py:440-475
Backend Implementations
Redis Caching
The RedisCache implementation supports standard Redis and Redis Cluster configurations. It utilizes Redis' native TTL features to expire keys. It also includes batch optimization through a redis_flush_size parameter, allowing the system to buffer writes in memory before flushing to Redis in high-traffic scenarios.
Sources: litellm/caching/redis_cache.py, litellm/caching/caching.py:101-118
S3 Caching
The S3Cache uses boto3 to store cached objects in a specified S3 bucket. Unlike memory-based caches, it provides long-term persistence across process restarts and distributed nodes without requiring a managed Redis instance.
Sources: litellm/caching/s3_cache.py, litellm/caching/caching.py:133-146
Dual Cache
The DualCache strategy is designed to minimize network latency. It checks InMemoryCache first (Tier 1). If a miss occurs, it checks RedisCache (Tier 2). If found in Redis, it populates the local memory cache to speed up subsequent identical requests.
Configuration and Cache Keys
Key Generation Logic
Cache keys are generated by concatenating relevant API parameters into a string and hashing them. This ensures that different configurations (e.g., changing temperature) result in different keys.
Included Parameters:
model(ormodel_groupif using Router)messages/promptinput(for embeddings)- Optional provider-specific params (if
enable_caching_on_provider_specific_optional_paramsis True)
Sources: litellm/caching/caching.py:186-213, litellm/caching/caching.py:246-267
Namespacing
Users can isolate cache entries by providing a namespace. This prefix is prepended to the hashed key (e.g., org_123:hash). Namespaces can be configured globally in the Cache object or dynamically per request.
Sources: litellm/caching/caching.py:344-365, tests/local_testing/test_caching.py:1191-1250
Cache Controls (TTL and Max-Age)
LiteLLM supports several mechanisms for controlling cache expiration:
| Parameter | Source | Description |
|---|---|---|
ttl | Global/Request | Time-to-live in seconds for the backend (e.g., Redis EX). |
s-maxage | Request | Max age allowed for a cached response; returns None if older. |
no-cache | Request | Bypasses cache check and forces a provider call. |
Sources: litellm/caching/caching.py:440-475, litellm/caching/caching.py:397-405, tests/local_testing/test_caching.py:1010-1044
Summary
The LiteLLM caching system provides a flexible, tiered approach to reducing LLM latency and cost. By supporting local memory, Redis, and S3, it scales from simple scripts to large-scale distributed systems. The inclusion of DualCache and SemanticCache further optimizes hit rates and access times for high-performance AI applications. Consistent SHA-256 key generation ensures that cache hits are accurate and parameter-sensitive.