Memory Creation & Structuring
Relevant source files
The following files were used as context for generating this wiki page:
- backend/utils/prompts.py
- desktop/macos/Backend-Rust/src/services/firestore/memories_repository.rs
- backend/utils/llm/chat.py
- desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/MemoryExtraction/MemoryAssistantSettings.swift
- backend/utils/llm/external_integrations.py
- desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift
- app/e2e/flows/add-edit-memory.yaml
Memory Creation & Structuring is the core system within Omi responsible for transforming raw data—such as conversation transcripts, screenshots, and manual notes—into high-quality, actionable, and structured "memories." This process filters out mundane or trivial information to ensure that only significant facts about the user or valuable external wisdom are preserved for long-term use by the AI mentor.
The system employs advanced LLM prompting strategies, strict categorization tests, and structured data schemas to maintain a clean and relevant knowledge base. These memories serve as the primary context for personalizing AI responses, generating daily summaries, and grounding the agent's understanding of the user's digital life.
Memory Extraction Logic
The extraction process follows a rigorous multi-step workflow designed to ensure high confidence and data quality. The system distinguishes between different sources of information, such as voice conversations or visual screen context.
The Categorization Test
Every potential memory must pass a specific decision tree to determine its validity and type.
- Interesting Memories (External Wisdom): Actionable advice, frameworks, or strategies originating from external sources (people, books, articles). They require clear attribution (e.g., "Source: Insight").
- System Memories (User Facts): Facts about the user’s own preferences, realizations, network, projects, and commitments.
Sources: backend/utils/prompts.py:44-54, desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/MemoryExtraction/MemoryAssistantSettings.swift:42-52
Workflow Architecture
The following diagram illustrates the logical flow of memory extraction from a conversation or screenshot:
Data Models and Schemas
Memories are stored with extensive metadata to allow for complex filtering, scoring, and source attribution.
Memory Data Structure
The Rust-based Firestore repository defines a MemoryDB structure containing fields for both the content and the context of extraction.
| Field | Type | Description |
|---|---|---|
id | String | Unique identifier (often a seed from content hash) |
content | String | The actual text of the memory (max 15 words preferred) |
category | Enum | system, interesting, manual, or specific sub-categories |
scoring | String | Used for ranking memories by relevance/recency |
source | String | Origin: desktop, omi, screenshot, or phone |
source_app | String | The application active during extraction |
confidence | Double | AI confidence score (0.0 - 1.0) |
tags | Array | Semantic tags (e.g., core, work, skills) |
Sources: desktop/macos/Backend-Rust/src/services/firestore/memories_repository.rs:118-150, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:462-484
Category and Tag Definitions
For structured fact extraction (often used in background indexing), the system categorizes facts into specific domains.
| Category | Tags | Focus Area |
|---|---|---|
| core | core | Age, residence, marital status, health. |
| work | work | Occupation, professional life, job details. |
| skills | skills | Abilities and expertise (e.g., Python, Java). |
| learnings | life_lessons, world_facts | General wisdom or insights about the world. |
Sources: backend/utils/prompts.py:168-200
Implementation Details
LLM Prompting Strategy
The system uses ChatPromptTemplate to enforce strict formatting and exclusion rules. For example, memories must be under 15 words and avoid "banned language" such as filler phrases ("indicates a...") or transient verbs ("is working on").
# Example of strict formatting requirements
FORMAT_REQUIREMENTS = """
• Maximum 15 words per memory (strict limit)
• Start with {user_name} when the memory is about them
• Focus on "who" and "what", not "when" (Timelessness)
• Use actual names with >90% confidence
"""
Sources: backend/utils/prompts.py:118-132, backend/utils/prompts.py:146-155
Manual Memory Management
Users can also curate their own knowledge base through manual creation and editing. This involves a UI-driven flow to add, edit, or soft-delete memories with an undo window.
Deduplication and Updates
To prevent redundancy, the system performs a "Consolidation Check" before saving. If a new memory contradicts an existing one (e.g., "Works at Google" vs. "Left Google for OpenAI"), it is added as an update. If it is semantically identical, it is rejected.
Sources: backend/utils/prompts.py:100-116, desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/MemoryExtraction/MemoryAssistantSettings.swift:101-105
Conclusion
Memory Creation & Structuring in Omi is a sophisticated pipeline that ensures user-specific data is distilled into a meaningful format. By combining automated extraction from active context with manual curation capabilities, Omi builds a reliable and high-fidelity knowledge graph that powers its mentor-like persona.
Sources: backend/utils/prompts.py, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift