Caching Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• backend/advanced_cache_system.py
• ml/redis_cache.py
• IMPLEMENTATION_SUMMARY.md
• A_PLUS_UPGRADE_PLAN.md
• DOCKER_MCP_DEPLOYMENT.md
• QUICK_REFERENCE.md
• PRODUCTION_CHECKLIST.md

The DevSkyy platform implements a sophisticated, multi-layer caching strategy to enhance performance, reduce API call costs, and ensure scalability. The system primarily uses a combination of an in-memory LRU (Least Recently Used) cache for immediate access and a distributed Redis cache for persistence and sharing across services. This strategy is crucial for achieving sub-100ms response times and significant cost savings, particularly for repeated queries and ML model inferences.

The caching system is designed to be asynchronous, fitting seamlessly into the FastAPI and asyncio-based architecture of the project. It includes a fallback mechanism, ensuring that if the Redis service is unavailable, the system can degrade gracefully to use the in-memory cache alone.

Sources: QUICK_REFERENCE.md, backend/advanced_cache_system.py, IMPLEMENTATION_SUMMARY.md

Architecture

The caching architecture is a two-layer system designed for speed and resilience. A request for data first checks a local, in-memory cache, then a shared Redis cache, and finally fetches from the source (e.g., database, external API) if not found in either cache.

Sources: backend/advanced_cache_system.py, QUICK_REFERENCE.md

Data Flow

The following diagram illustrates the data retrieval flow through the caching layers.

This diagram shows the hierarchical lookup process, prioritizing the fastest cache (in-memory) first.

Sources: backend/advanced_cache_system.py:38-68, A_PLUS_UPGRADE_PLAN.md

Components

Component	Description	Implementation File
AdvancedCacheSystem	The core class managing the multi-layer cache, including the in-memory LRU cache and the Redis client. It handles the logic for get, set, and delete operations across both layers.	backend/advanced_cache_system.py
CacheManager	An async-native class that provides a direct interface to Redis for caching operations like get, set, delete, and pattern-based clearing. It uses redis.asyncio for non-blocking I/O.	A_PLUS_UPGRADE_PLAN.md
RedisCache (ML)	A specialized Redis cache implementation for ML models and results, demonstrating a domain-specific use of the caching infrastructure.	ml/redis_cache.py
Redis Service	The distributed caching backend, deployed as a Docker container. It provides high-performance storage for sessions and cached data.	DOCKER_MCP_DEPLOYMENT.md

Sources: backend/advanced_cache_system.py, A_PLUS_UPGRADE_PLAN.md, ml/redis_cache.py, DOCKER_MCP_DEPLOYMENT.md

Redis Integration

Redis serves as the primary distributed caching layer. The system interacts with Redis through a dedicated CacheManager class that leverages redis.asyncio for asynchronous operations.

CacheManager Class

The CacheManager class provides a clean, async-first interface for interacting with the Redis cache.

This diagram shows the main methods of the CacheManager for Redis operations.

Sources: A_PLUS_UPGRADE_PLAN.md

The implementation uses pickle to serialize and deserialize Python objects, allowing complex data structures to be stored in Redis.

# A_PLUS_UPGRADE_PLAN.md

import redis.asyncio as redis
from typing import Any, Optional
import pickle

class CacheManager:
    def __init__(self, redis_url: str = "redis://localhost:6379"):
        self.redis = redis.from_url(redis_url, decode_responses=False)

    async def get(self, key: str) -> Optional[Any]:
        """Get cached value"""
        value = await self.redis.get(key)
        if value:
            return pickle.loads(value)
        return None

    async def set(self, key: str, value: Any, ttl: int = 300):
        """Set cached value with TTL (default 5 min)"""
        await self.redis.set(key, pickle.dumps(value), ex=ttl)

Sources: A_PLUS_UPGRADE_PLAN.md

Cache Key Structure

A standardized key structure is used to ensure consistency and avoid collisions.

devskyy:{data_type}:{identifier}

Examples:

• devskyy:query:{query_type}:{hash(filters)}
• devskyy:health:status
• devskyy:agents:directory

This structured approach allows for features like pattern-based cache clearing (e.g., clearing all keys related to a specific agent).

Sources: IMPLEMENTATION_SUMMARY.md

Caching Strategies

The system employs different strategies based on the data type and access patterns. The primary strategies are LRU for the in-memory cache and TTL-based eviction for Redis.

Time-to-Live (TTL) Strategy

Different types of data are cached with varying TTLs to balance performance with data freshness.

Data Type	TTL	Cache Hit Rate (Est.)	Purpose
Agent List	1 hour	95%	Reduce load on agent registry.
Health Status	30 seconds	90%	Provide fast health check responses.
Metrics	5 minutes	85%	Cache frequently accessed performance data.
ML Predictions	5 minutes	70%	Avoid re-running expensive ML models.

Sources: IMPLEMENTATION_SUMMARY.md

ML Model Caching

A dedicated RedisCache class is used within the machine learning modules. This class is tailored for caching ML model objects and their prediction results, which is critical for reducing latency and computational cost.

# ml/redis_cache.py

class RedisCache:
    def __init__(self, host='localhost', port=6379, db=0):
        # ... connection logic ...

    def get_model(self, model_name):
        # ... logic to get a serialized model ...

    def set_model(self, model_name, model, ex=3600):
        # ... logic to serialize and store a model ...

    def get_prediction(self, cache_key):
        # ... logic to get a cached prediction ...

    def set_prediction(self, cache_key, prediction, ex=300):
        # ... logic to store a prediction result ...

This specialized class highlights how the caching framework is adapted for specific domains within the application.

Sources: ml/redis_cache.py

Performance and Cost Impact

The caching strategy has a significant positive impact on both application performance and operational costs. By serving requests from the cache, the system avoids expensive API calls and database queries.

API Request Flow with Caching

The sequence diagram below shows how a typical API request is handled, prioritizing cache lookups.

Cost and Performance Metrics

Metric	Before Caching (v1)	After Caching (v2)	Savings
Response Time (Cache Hit)	N/A	<10ms	-
Average Token Usage	~488 tokens/req	~116 tokens/req	75%
API Cost (1k req/day)	$4.50/day	$0.20/day	95.6%
Average Cache Hit Rate	N/A	85%	-

These metrics demonstrate that the caching layer is highly effective, leading to an estimated 90-96% reduction in API costs and a significant improvement in response latency.

Sources: IMPLEMENTATION_SUMMARY.md

Configuration and Deployment

The Redis cache is a core part of the production infrastructure and is configured via environment variables.

Environment Variables

The connection to the Redis server is managed using a REDIS_URL.

Variable	Example Value	Description
REDIS_URL	redis://redis:6379/0	The connection string for the Redis instance.
REDIS_PASSWORD	change-in-production	Password for Redis authentication.

Sources: DOCKER_MCP_DEPLOYMENT.md

Docker Deployment

In a production environment, Redis is deployed as a dedicated service using Docker Compose. The application services then connect to this Redis container.

# In docker-compose.mcp.yml

services:
  devskyy-mcp:
    # ...
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    restart: unless-stopped

volumes:
  redis-data:

This configuration ensures a persistent and reliable caching service for the entire application stack.

Sources: DOCKER_MCP_DEPLOYMENT.md

Summary

The caching strategy in DevSkyy is a cornerstone of its enterprise-grade architecture. By employing a multi-layer approach with an in-memory LRU cache and a distributed Redis backend, the system achieves significant improvements in performance, cost-efficiency, and scalability. The strategy is well-integrated across the platform, with specialized implementations for different domains like machine learning, and is configured for robust production deployment. The result is a fast, responsive, and economically viable platform capable of handling high-load scenarios.

Sources: IMPLEMENTATION_SUMMARY.md, A_PLUS_UPGRADE_PLAN.md, QUICK_REFERENCE.md