Storage Management
Relevant source files
The following files were used as context for generating this wiki page:
Storage Management in the FLEXT API project provides a structured, model-driven approach to handling ephemeral data, caching, and runtime state. The system is designed around a simple in-memory storage implementation that uses Pydantic 2 models for validation and metadata tracking, ensuring that stored entries remain type-safe and consistent with domain requirements.
The module scope encompasses internal metadata management (such as TTL and timestamps), runtime statistics tracking (hits, misses, and operation counts), and configuration through namespaced settings. This architecture allows the API to maintain high-performance access to transient data while providing clear observability into storage health and usage.
Sources: src/README.md, src/flext_api/_models/storage.py:1-15, examples/01_basic_usage.py:101-125
Architecture and Data Models
The storage architecture is organized into a specific namespace under the m.Api.Storage shard. It uses a centralized state model to manage a collection of metadata-wrapped entries.
Storage Components Hierarchy
The following diagram illustrates the relationship between different storage models within the FlextApiModelsStorage class.
The storage system relies on State as the central runtime container, while Stats provides a read-only snapshot of performance metrics.
Sources: src/flext_api/_models/storage.py:18-125
Data Model Definitions
| Model | Description | Key Fields |
|---|---|---|
| Settings | Configuration for storage instances. | namespace, backend, max_size, default_ttl |
| Metadata | Wrapper for stored values including lifecycle data. | value, timestamp, ttl, created_at |
| State | The active runtime state of a storage instance. | entries, operations_count, cache_hits, cache_misses |
| Stats | Calculated performance and usage metrics. | storage_size, memory_usage, hit_ratio |
Sources: src/flext_api/_models/storage.py:23-112
Lifecycle and Expiration Logic
Entries in the storage system are governed by Time-To-Live (TTL) logic. The Metadata model includes an expired property that determines if a record is still valid based on its monotonic creation timestamp and defined TTL.
This logic ensures that stale data can be identified and evicted from the entries mapping during runtime operations.
Sources: src/flext_api/_models/storage.py:65-79
Storage Utilities and Integration
Storage management is integrated into the broader API through specialized utilities and typing adapters. The project utilizes u.TypeAdapter for strict validation of storage entries and JSON mappings.
Type Adapters
The following adapters are defined for storage-related validation:
STORAGE_ENTRY_ADAPTER: Validatest.JsonMappingfor storage entries.API_JSON_VALUE_ADAPTER: Validates generalt.JsonValueused in storage payloads.
Sources: src/flext_api/typings.py:84-110
Implementation Example
A typical storage flow involves initializing settings, creating metadata-wrapped entries, and updating the state:
# Create metadata for a value
entry = m.Api.Storage.Metadata(
value=response_body,
timestamp=u.generate_iso_timestamp(),
ttl=3600
)
# Initialize state with the entry
state = m.Api.Storage.State(
entries={"key": entry},
operations_count=1
)
Sources: examples/01_basic_usage.py:108-124
Observability and Statistics
The Stats model provides a mechanism to monitor the efficiency of the storage system. It calculates a hit_ratio to evaluate cache performance and tracks estimated memory usage.
Statistics Calculation
| Metric | Calculation / Source |
|---|---|
| Total Operations | Sum of hits and misses (tracked in State). |
| Storage Size | Number of keys present in the entries mapping. |
| Hit Ratio | cache_hits / total_operations (returns 0.0 if operations are 0). |
| Memory Usage | Estimated based on the representation length of the entries. |
Sources: src/flext_api/_models/storage.py:118-132, examples/01_basic_usage.py:126-135
Conclusion
Storage Management within the FLEXT API provides a robust framework for handling in-memory data with strict Pydantic-based validation. By combining detailed metadata tracking, automatic expiration logic, and comprehensive statistics, the system ensures that transient state is managed predictably and efficiently. This module serves as a critical component for caching API responses and maintaining runtime stability across the library.