Data and Persistence Layer
Relevant source files
The following files were used as context for generating this wiki page:
The data and persistence layer in monGARS is a modular and multi-tiered system designed for resilience, performance, and maintainability. It combines a primary SQL database for durable storage with multiple caching layers to ensure fast data retrieval for conversational context and other frequent operations. The architecture is built around a central repository pattern, ensuring that all data access is consistent and managed.
The core components include a PostgreSQL database with schemas managed by SQLModel, a central PersistenceRepository for all database interactions, and a sophisticated caching strategy that includes the in-memory Hippocampus module, Redis, and disk-based caches. Schema evolution is formally handled by Alembic, ensuring that database changes are version-controlled and repeatable.
Sources: README.md, monGARS/AGENTS.md
Architecture Overview
The persistence architecture is designed to decouple application logic from the underlying data storage mechanisms. Services across the application do not interact with the database directly; instead, they go through a set of well-defined abstractions like the PersistenceRepository and caching services.
This diagram illustrates the high-level flow of data from application services to the storage backends.
This diagram shows the layered architecture of the persistence system, with services accessing data through an abstraction layer.
Sources: README.md, monGARS/AGENTS.md, configs/agents/agents_config.json
Primary Storage: SQL Database
The primary durable storage for monGARS is a PostgreSQL database. It is used to store core application data, including user accounts, personality profiles, and conversation history.
Sources: README.md:83-83
ORM and Schema Definition
The project uses SQLModel as its Object-Relational Mapper (ORM). SQLModel combines Pydantic and SQLAlchemy, providing a modern, type-hinted way to define database models. These models are the basis for the database schema.
Key characteristics:
- Technology: PostgreSQL is the specified SQL backend.
- ORM: SQLModel is used for defining data tables.
- Initialization: The
init_db.pyscript is responsible for preparing the database, including applying migrations and creating tables.
Sources: README.md:58-58, README.md:83-83, monGARS/AGENTS.md:30-30, ROADMAP.md:46-46
Schema Migrations with Alembic
Database schema changes are managed authoritatively by Alembic. This ensures that schema evolution is version-controlled, repeatable, and separated from the application code.
- All migration scripts are located in the
alembic_migrations/directory. - The
init_db.pyscript is used to apply these migrations when setting up the environment. - The project roadmap emphasizes extending Alembic migrations for any new SQLModel tables, reinforcing this as a strict development practice.
The migration workflow is a critical part of the deployment and development process.
This diagram shows the developer workflow for managing database schema changes using SQLModel and Alembic.
Sources: README.md:83-83, monGARS/AGENTS.md:30-30, ROADMAP.md:46-46
Data Access Layer
To maintain separation of concerns and ensure consistent data handling, all database access is routed through a central repository.
PersistenceRepository
The core.persistence.PersistenceRepository is the designated gateway for all database operations. Development guidelines mandate its use for any interaction with the SQL database, preventing direct queries from other parts of the application. This pattern centralizes data access logic, making it easier to manage, test, and optimize.
Sources: monGARS/AGENTS.md:30-30, configs/agents/agents_config.json:71-71
Caching Strategy
monGARS employs a multi-tiered caching strategy to optimize performance and reduce database load. The system blends in-memory, Redis, and disk-based caching.
The key components of the caching layer are:
Hippocampus: An in-memory caching module responsible for holding short-term conversational context.- Redis & Disk Caching: Tiers used for more persistent caching needs.
IrisModule Cache: A concrete example of in-memory caching for web retrieval.
Sources: README.md:29-30, monGARS/AGENTS.md:32-32
Iris Caching Implementation
The monGARS.core.iris.py module provides a clear, practical implementation of a resilient, in-memory caching system for its web retrieval functions. The Iris class uses two internal caches to store results from web searches and fetched documents.
This implementation serves as a blueprint for caching within the monGARS ecosystem.
This class diagram shows the key components of the Iris class, including its internal caches and public methods.
Sources: monGARS/core/iris.py
Cache Logic
The Iris cache is an in-memory, time-aware LRU (Least Recently Used) cache implemented using Python's collections.OrderedDict.
- Two-Level Cache: It maintains separate caches for search query results (
_search_cache) and fetched web documents (_document_cache). - TTL (Time-To-Live): Each cache entry has a configurable TTL. Expired entries are evicted upon access.
- LRU Eviction: When a cache exceeds its configured size, the least recently used items are removed.
- Thread Safety: Asynchronous locks (
asyncio.Lock) are used to prevent race conditions when accessing the caches. - In-flight Request Coalescing: It tracks in-flight requests for documents to avoid fetching the same URL multiple times concurrently.
The sequence diagram below details the logic for a fetch_document call.
This sequence diagram illustrates the caching and request coalescing logic within the Iris.fetch_document method.
Sources: monGARS/core/iris.py:151-171, monGARS/core/iris.py:302-316, monGARS/core/iris.py:643-698
Configuration
The Iris cache behavior is configurable via its constructor.
| Parameter | Type | Description |
|---|---|---|
max_concurrency | int | Maximum number of concurrent HTTP requests. |
request_timeout | float | Timeout in seconds for each HTTP request. |
max_retries | int | Number of retries for failed requests. |
search_cache_ttl | float | TTL in seconds for search results in the cache. |
search_cache_size | int | Maximum number of items in the search cache. |
document_cache_ttl | float | TTL in seconds for documents in the cache. |
document_cache_size | int | Maximum number of items in the document cache. |
orchestrator_result_limit | int | Number of search results to fetch from the orchestrator. |
Sources: monGARS/core/iris.py:92-148
Summary
The data and persistence layer of monGARS is a well-architected system that prioritizes both performance and long-term maintainability. By mandating the use of the PersistenceRepository and Alembic, it establishes clear and robust patterns for data management. The multi-tiered caching strategy, exemplified by the Iris module, demonstrates a commitment to performance and resilience by reducing latency and protecting against external service failures. This combination of durable SQL storage and intelligent caching makes the persistence layer a solid foundation for the entire application.
Sources: README.md, monGARS/AGENTS.md