AI Memory System
Relevant source files
The following files were used as context for generating this wiki page:
The AI Memory System in ClassMate is a sophisticated dual-layered architecture designed to provide both short-term contextual awareness for agents and long-term semantic retrieval of lecture content. It enables the Study Assistant and Note Orchestrator to maintain state, learn from historical data, and provide personalized academic support through vector-based indexing and event tracking.
The system is primarily composed of the Vector Memory Store, which handles long-term storage of embeddings for semantic search, and the Agent Memory, which tracks transient events during a processing session. This architecture ensures that AI agents can "remember" previous lecture details, summarize content effectively, and answer user queries with high relevance.
Sources: ai_backend/agents/study_assistant.py, ai_backend/agents/vector_store.py
Vector Memory Store Architecture
The VectorMemoryStore serves as the long-term repository for all AI-generated content and lecture transcripts. It is designed with a hybrid approach to ensure compatibility across different environments, utilizing pgvector on PostgreSQL for production-grade semantic search and falling back to keyword-based matching on SQLite for development or local deployments.
Sources: ai_backend/agents/vector_store.py:27-44
Data Models and Storage
The system stores information in a VectorDocument model. Each document is associated with a specific user and session, allowing for filtered searches and privacy.
| Field | Type | Description |
|---|---|---|
id | String (UUID) | Unique identifier for the document. |
user_id | String | Reference to the owner (indexed for performance). |
session_id | String | Reference to the specific lecture session. |
content | Text | The raw text (transcript, summary, or note). |
content_type | String | Category: transcript, summary, note, or flashcard. |
embedding | Vector(1536) | pgvector column for 1536-dimension embeddings. |
Sources: ai_backend/agents/vector_store.py:20-27, ai_backend/agents/vector_store.py:58-69
Semantic Search Flow
The system utilizes cosine similarity (via the <=> operator in pgvector) to find content that is semantically similar to a user's query or a specific concept.
The diagram shows the decision logic between high-performance vector search and keyword-based fallback.
Sources: ai_backend/agents/vector_store.py:112-140
Agent Memory and Event Tracking
The AgentMemory (integrated with the NotesOrchestrator and various agents) provides transient context during the execution of a task. It tracks "events" to ensure agents like the RefinerAgent can adjust outputs based on the actions of previous agents like the EvaluatorAgent.
Memory Event Types
Agents register their actions in the memory to create an audit trail and context for later refinement.
| Event Name | Agent Source | Purpose |
|---|---|---|
summary_generated | SummaryAgent | Tracks the length of the initial summary. |
key_points_generated | KeyPointsAgent | Tracks the count of identified main concepts. |
evaluation_done | EvaluatorAgent | Logs issues like "empty_summary" or "too_many_points". |
refiner_applied | RefinerAgent | Logs the specific corrections made to the notes. |
Sources: ai_backend/agents/note_agents.py:16, ai_backend/agents/note_agents.py:61, ai_backend/agents/note_agents.py:76
Memory Flow in Note Generation
This sequence demonstrates how Agent Memory acts as the communication bus between specialized AI agents.
Sources: ai_backend/agents/note_agents.py:64-100
Study Assistant Integration
The StudyAssistantAgent utilizes the Memory System to perform complex academic tasks. It leverages VectorMemoryStore to fetch relevant lecture snippets for its ask_question and explain_concept functions.
Key API Endpoints
The Memory System is exposed through several AI-specific endpoints.
| Endpoint | Request Model | Description |
|---|---|---|
/api/v1/ai/ask | AskQuestionRequest | RAG-based Q&A using vector search for context. |
/api/v1/ai/index | IndexRequest | Manually indexes transcripts or summaries into memory. |
/api/v1/ai/search | SearchRequest | Performs semantic search across all indexed user content. |
Sources: ai_backend/api/ai_agent_api.py:46-60, ai_backend/api/ai_agent_api.py:110-140
Semantic Indexing Implementation
When a transcript or summary is indexed, the system generates an embedding (typically using OpenAI's text-embedding-3-small) and stores it in the vector database.
def index_transcript(self, user_id, session_id, transcript, metadata=None):
embedding = None
embeddings_model = self._ensure_embeddings()
if embeddings_model:
embedding = embeddings_model.embed_query(transcript[:2000])
return self._vector_store.store_document(
user_id=user_id,
session_id=session_id,
content=transcript,
content_type="transcript",
embedding=embedding
)
Sources: ai_backend/agents/study_assistant.py:220-240
Critical Limitations and Infrastructure
Analysis of the implementation reveals several architectural constraints and production requirements.
- Memory Volatility: The
AgentMemoryused during processing is often stored in-process (e.g.,_session_memorydict), meaning a worker crash results in context loss. - Database Locking: While PostgreSQL is supported, the default configuration often points to SQLite, which lacks concurrent write support for multi-worker AI processing.
- Embedding Limits: The vector store currently uses the first 2,000 characters of a transcript for embedding generation, which may miss context in longer lectures.
- Encryption: Security audits indicate that while the
VectorDocumentmodel exists, sensitive content may be stored in plaintext unlessEncryptionServiceis explicitly invoked.
Sources: GAPS_IMPLEMENTATION_SUMMARY.md:Section 5, ai_backend/agents/vector_store.py:65
Conclusion
The AI Memory System provides the fundamental "knowledge base" for the ClassMate application. By combining transient agent events with long-term pgvector storage, it enables sophisticated features like context-aware tutoring and self-correcting note generation. However, production deployments must ensure PostgreSQL is utilized to move beyond simple keyword fallbacks and provide the high-performance semantic search intended by the architecture.