Vector DB & Retrieval Augmented Generation (RAG)
Relevant source files
The following files were used as context for generating this wiki page:
The Vector DB and Retrieval Augmented Generation (RAG) system in Omi provides the AI assistant with long-term memory and context-aware reasoning capabilities. By leveraging a vector database (Pinecone) alongside structured local data (SQLite/Firestore), the system allows the assistant to retrieve relevant past conversations, user facts, and digital activity to provide personalized responses.
This system is core to the "2nd brain" mission, enabling the agent to go beyond simple chat by grounding its output in the user's actual life history, including transcribing meetings, screen context, and personal preferences.
System Architecture
The RAG pipeline operates through a multi-layered retrieval strategy that combines semantic search with structured filtering. The backend uses Pinecone for high-dimensional vector embeddings, typically configured with 1536 dimensions for OpenAI embeddings.
Data Flow for Retrieval
The following diagram illustrates how a user query is processed to retrieve relevant context.
Vector Database Integration
Omi utilizes Pinecone as the primary vector database for semantic search. The backend requires a PINECONE_INDEX_NAME and an index created with 1536 dimensions to support OpenAI embeddings.
Metadata Extraction
Before data is stored in the vector DB, it undergoes a transformation where an LLM extracts relevant metadata fields to enable precise filtering later.
| Field | Description |
|---|---|
| People | Names mentioned during conversations or in documents. |
| Topics | Main subjects and subtopics discussed. |
| Entities | Products, technologies, places, or organizations. |
| Dates | Specific time references extracted and normalized to YYYY-MM-DD. |
Sources: backend/utils/llm/chat.py:539-552, backend/README.md:95-103
Retrieval Strategies
The system employs different strategies based on whether the user is asking about a static fact or a specific event.
Fact vs. Event Distinction
- Fact/Preference: Uses
get_memories_tool. Covers static information like a user's favorite food, age, or relationships. - Event/Incident: Uses
search_conversations_tool. Covers specific occurrences like "When did I meet John?" or "What happened at the party?"
Tool-Based Retrieval Flow
The agentic RAG prompt guides the LLM to select the appropriate tool based on the query type.
Prompt Engineering & Citations
A critical requirement of the RAG system is technical accuracy and the prevention of hallucinations. The assistant is strictly instructed to say "I don't know" if tools return empty results.
Citing Instructions
When context is retrieved from memories or conversations, the AI must provide inline citations:
- Citations appear at the end of each sentence containing retrieved info.
- Format:
[index]immediately after the sentence with no space (e.g.,yesterday[1][2].). - Citations are only permitted for
<memories>, not for<user_facts>or<previous_messages>.
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:96-105, backend/utils/llm/chat.py:279-286
Technical Configuration
Environment Variables (Backend)
The following variables are required to be set in the .env file for the RAG system to function:
| Variable | Requirement |
|---|---|
PINECONE_INDEX_NAME | Name of the index in the Pinecone Console. |
OPENAI_API_KEY | Used for generating embeddings and responding to queries. |
ADMIN_KEY | Used for local development authentication bypass. |
Memory Management (Backend)
The backend implements strict memory management for processing large amounts of transcript data, including:
- Deleting byte arrays after processing.
- Clearing dictionaries and lists that hold temporary retrieval data.
- Offloading blocking database/vector calls to
db_executorto avoid blocking the async event loop.
Sources: backend/README.md:95-103, AGENTS.md:55-70, backend/AGENTS.md:145-160
Conclusion
The Vector DB and RAG system serves as the foundational intelligence layer for Omi. By integrating Pinecone for semantic retrieval and strictly enforcing citation-based responses through specialized prompts, the system ensures that the AI assistant remains a reliable and personalized extension of the user's memory.