AI Chat & Context Agents
Relevant source files
The following files were used as context for generating this wiki page:
The AI Chat and Context Agents system in Omi serves as a "second brain," providing users with a personalized assistant that retains and utilizes information captured from screen activity and real-time conversations. The system integrates Large Language Models (LLMs) with long-term memory (facts and habits) and short-term context (recent screenshots and transcripts) to deliver proactive insights and conversational support across desktop, mobile, and wearable platforms.
Sources: README.md:5-9, backend/utils/llm/chat.py:328-335
Architecture and Interaction Flow
The system operates through a specialized pipeline that classifies user intent, retrieves relevant context from various databases (Firestore, Redis, SQLite), and generates responses using tailored prompts. It distinguishes between general conversational queries, requests for personal data, and administrative tasks within the Omi ecosystem.
Intent Detection and Context Requirements
Before generating a response, the system determines if the user's query requires external context or if it is a simple conversational greeting. It specifically identifies if a question is about the Omi product itself or the user's personal memories.
The logic differentiates between "Product Questions" (e.g., "How does Omi work?") and "Personal Data Questions" (e.g., "What did I say yesterday?"). Sources: backend/utils/llm/chat.py:118-124, backend/utils/llm/chat.py:145-165, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:657-700
Memory Extraction and Management
The "Context Agents" include specialized modules for extracting "timeless" memories from raw transcripts. These agents distinguish between facts about the user and valuable external wisdom.
Memory Categories
| Category | Description | Source Requirement |
|---|---|---|
| SYSTEM | Facts about the user: preferences, opinions, realizations, and network relationships. | Internal to the user. |
| INTERESTING | External wisdom, frameworks, or advice learned from others. | Must have clear attribution. |
Sources: backend/utils/prompts.py:44-60, backend/utils/prompts.py:84-100
Extraction Workflow
- Identify Speakers: Resolve names or roles (e.g., "colleague") instead of generic speaker IDs.
- Apply Categorization Test: Determine if the information is wisdom from others or a fact about the user.
- Filter Quality: Remove trivial details (e.g., "likes coffee") or transient info (e.g., "meeting at 2 PM").
- Deduplication: Compare against existing memories to prevent redundant entries.
Sources: backend/utils/prompts.py:32-42, backend/utils/prompts.py:146-160
Desktop Contextual Agents
On the macOS desktop client, agents have access to local SQLite databases containing screen history (OCR), application usage, and local file metadata.
SQL-Driven Retrieval Strategies
The desktop agent uses specific SQL patterns to gather context from the user's activity:
-- Search screenshot OCR text via FTS5
SELECT s.* FROM screenshots s
JOIN screenshots_fts ON screenshots_fts.rowid = s.id
WHERE screenshots_fts MATCH 'keyword'
-- Recent app usage tracking
SELECT appName, COUNT(*) as count
FROM screenshots WHERE timestamp >= datetime('now', 'start of day')
GROUP BY appName ORDER BY count DESC
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:340-360, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:510-525
Onboarding Agent
The Onboarding Agent is a specialized state machine designed to build a user's initial knowledge graph. It performs:
- File Scans: Analyzes
~/Documents,~/Developer, and~/Downloadsfor project types. - Web Research: Researches the user's company or public projects to personalize the experience.
- Permission Management: Conversationally requests macOS permissions (Microphone, Accessibility, etc.).
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:386-420, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:490-505
Tool-Assisted Response Generation (RAG)
The system utilizes an Agentic QA prompt that empowers the LLM to call various tools for data retrieval. This process is strictly governed by accuracy rules to prevent hallucinations.
Core Retrieval Tools
| Tool | Purpose |
|---|---|
search_conversations_tool | Semantic search for specific events or incidents. |
get_conversations_tool | Chronological views and time-based activity queries. |
get_memories_tool | Retrieval of static facts, preferences, and relationships. |
manage_daily_summary_tool | Controls for user notifications and reflection times. |
Sources: backend/utils/llm/chat.py:425-445, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:186-210
Accuracy Rules
- No Fabrication: If tools return empty results, the agent must state it does not have the info in 1-2 lines.
- Human Tone: Agents are instructed to avoid robotic phrases like "in the logs" or "according to the data," instead using "I don't remember" or "I don't have that."
- Timezone Awareness: All calculations are performed in the user's local timezone before being converted to ISO for tool parameters.
Sources: backend/utils/llm/chat.py:450-465, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:237-260
Integration with External Plugins
Context agents can be extended via plugins (e.g., Slack, GitHub, Hive). These integrations allow the AI to move from passive observation to active task execution.
Conclusion
The AI Chat & Context Agents system is a multi-layered architecture that transforms raw sensor data (audio/screen) into a structured memory graph. By combining intent classification, rigorous memory extraction rules, and a tool-augmented conversational interface, Omi provides a proactive assistant that acts as a personalized extension of the user's memory.
Sources: backend/utils/llm/chat.py:408-415, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:275-285