Vector Store & RAG

Relevant source files

The following files were used as context for generating this wiki page:

• ai_backend/agents/vector_store.py
• ai_backend/agents/study_assistant.py
• ai_backend/api/ai_agent_api.py
• infra/docker/init.sql
• ai_workers/notes_worker.py
• PROJECT_STRUCTURE.md

The Vector Store and Retrieval-Augmented Generation (RAG) system in ClassMate provides the foundation for semantic search and intelligent AI interactions across lecture content. It enables the system to store embeddings of transcripts, summaries, and notes, allowing for context-aware Q&A, study guide generation, and concept explanations.

The system is designed with a hybrid approach: it utilizes pgvector on PostgreSQL for high-performance vector similarity searches while providing a keyword-based fallback for SQLite environments. This ensures that the AI agents, such as the StudyAssistantAgent, can always access relevant lecture context regardless of the underlying database infrastructure.
Sources: ai_backend/agents/vector_store.py:6-14, ai_backend/agents/study_assistant.py:10-15

Architecture and Data Flow

The Vector Store architecture revolves around the VectorMemoryStore class, which manages the lifecycle of document embeddings and retrieval logic. Documents are stored with associated metadata and high-dimensional vectors (1536 dimensions for OpenAI embeddings).

Data Ingestion and Retrieval Flow

The following diagram illustrates how lecture content is transformed into searchable vectors and retrieved by AI agents.

The flow begins with content ingestion where embeddings are generated and stored. During retrieval, a user query is embedded and compared against stored vectors to extract relevant snippets for the LLM.
Sources: ai_backend/agents/vector_store.py:48-90, ai_backend/agents/study_assistant.py:72-105

Database Schema

The core storage model is the VectorDocument, which persists the content and its mathematical representation.

VectorDocument Model

Field	Type	Description
id	String (UUID)	Primary key for the document.
user_id	String	Foreign key identifier for the user (indexed).
session_id	String	Optional identifier for the specific lecture session.
content	Text	The actual text content (transcript, note, etc.).
content_type	String	Type: transcript, summary, note, or flashcard.
metadata_json	Text	JSON string containing additional context.
embedding	Vector(1536)	The pgvector column for semantic search (PostgreSQL only).
created_at	DateTime	Timestamp of document creation.

Sources: ai_backend/agents/vector_store.py:22-35, ai_backend/agents/vector_store.py:59-65, infra/docker/init.sql:10-14

Semantic Search and RAG Logic

The StudyAssistantAgent implements RAG by performing a similarity search before invoking the Language Model (LLM). This ensures the AI's responses are grounded in specific lecture data.

Search Mechanisms

1. pgvector Search: Uses cosine similarity (<=> operator) to find the most relevant vectors. It calculates similarity as 1 - (embedding <=> :query_vector).
2. Keyword Fallback: If pgvector is unavailable, the system performs a standard SQL LIKE search on the content field.
3. Recent Search: If no query text or embedding is provided, the system returns the most recently indexed documents for that user.

Sources: ai_backend/agents/vector_store.py:112-140, ai_backend/agents/vector_store.py:151-170

Retrieval-Augmented Generation (RAG) Sequence

The sequence diagram below shows how the ai_agent_api coordinates with the store and assistant to answer questions.

API Endpoints

The AI Backend exposes several endpoints for managing and querying the vector store.

Endpoint	Method	Purpose
/api/v1/ai/index	POST	Manually index transcripts or summaries into the store.
/api/v1/ai/search	POST	Perform a semantic search across indexed lecture content.
/api/v1/ai/ask	POST	Ask a question that uses RAG to provide a context-aware answer.
/api/v1/ai/stats	GET	Retrieve statistics, such as the total count of indexed documents.

Sources: ai_backend/api/ai_agent_api.py:118-175

Implementation Details

Initialization and Fallback

The system detects the database type via the DATABASE_URL. If "postgresql" is present, it attempts to enable the vector extension and add a vector(1536) column to the table. If this fails or if SQLite is used, it sets _use_pgvector = False and relies on keyword matching.
Sources: ai_backend/agents/vector_store.py:44-67

Code Snippet: Storing a Vector

# ai_backend/agents/vector_store.py:84-90
if embedding and self._use_pgvector:
    emb_str = "[" + ",".join(str(x) for x in embedding) + "]"
    session.execute(
        text("UPDATE vector_documents SET embedding = :emb WHERE id = :id"),
        {"emb": emb_str, "id": doc_id}
    )

Integration with Workers

The NotesWorker triggers the generation of AI notes, which can then be indexed. The StudyAssistantAgent provides helper methods like index_transcript and index_summary to wrap the vector store logic and ensure embeddings are generated before storage.
Sources: ai_workers/notes_worker.py:42-55, ai_backend/agents/study_assistant.py:270-300

Summary

The Vector Store & RAG system transforms static lecture recordings into an interactive knowledge base. By combining pgvector for semantic retrieval and LangChain for LLM orchestration, ClassMate provides students with a powerful tool for querying their academic materials with high technical accuracy and contextual relevance.