macOS Desktop App & Capabilities
Relevant source files
The following files were used as context for generating this wiki page:
- desktop/macos/AGENTS.md
- AGENTS.md
- desktop/macos/Desktop/Sources/Chat/DesktopCapabilityRegistry.swift
- desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift
- desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/TaskExtraction/TaskAssistant.swift
- desktop/macos/Desktop/Sources/OnboardingPagedIntroCoordinator.swift
- desktop/macos/e2e/SKILL.md
The Omi macOS Desktop Application is a native Swift-based project designed to serve as an intelligent "second brain." It integrates real-time screen capture, transcription, and proactive assistance to help users manage tasks and memories directly from their computers. The architecture leverages a Swift Package Manager (SPM) structure, utilizing a local Rust backend for performance-heavy operations and a Python/Firebase cloud infrastructure for cross-device synchronization.
The app's primary mission is to capture, understand, and act upon the user's digital context. This is achieved through a suite of native capabilities, including semantic search over screen history, automated task extraction from messaging apps, and an agentic chat interface that can execute local SQL queries and manage background subagents.
Sources: README.md, desktop/macos/AGENTS.md:14-25, desktop/macos/e2e/SKILL.md:10-20
Architecture Overview
The desktop application is built using SwiftUI and Swift, operating as an SPM project without a traditional .xcodeproj file. It communicates with a local Rust backend (Backend-Rust) and an external Python API.
The architecture facilitates local-first data storage while maintaining cloud sync for mobility. The local omi.db database stores screenshots, OCR text, action items, and transcription segments, which are accessible to AI models via a structured capability registry.
Sources: desktop/macos/AGENTS.md:94-105, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:420-435, README.md:110-130
AI Capabilities & Tool Registry
Omi utilizes a DesktopCapabilityRegistry to expose specific functions (tools) to its agentic chat and proactive assistants. These tools are categorized by their latency (e.g., fastLocal, fastNetwork, asyncBackground) and the surfaces they support.
Key Capabilities Table
| Tool Name | Latency Class | Primary Purpose |
|---|---|---|
execute_sql | fastLocal | Runs SELECT/INSERT/UPDATE on local omi.db. |
semantic_search | fastLocal | Vector search over screen history history. |
get_daily_recap | fastLocal | Pre-formatted activity summary (apps, tasks, focus). |
spawn_agent | asyncBackground | Launches background subagents (e.g., OpenClaw, Hermes). |
search_conversations | fastNetwork | Semantic search across past voice conversations. |
get_task_agent_status | fastLocal | Inspects local task-chat agents and floating pills. |
Sources: desktop/macos/Desktop/Sources/Chat/DesktopCapabilityRegistry.swift:23-220
Local SQL Context
The execute_sql tool provides the LLM with direct access to structured data. The system prompt includes common SQL patterns for searching personal facts in memories, analyzing app usage in screenshots, and retrieving action_items.
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:330-365, desktop/macos/Desktop/Sources/Chat/DesktopCapabilityRegistry.swift:31-40
Proactive Task Extraction
The TaskAssistant is a core module that monitors the user's screen context to identify commitments and action items automatically. It specifically uses a "Fast-Path" for messaging applications like Slack, Discord, and Telegram.
Extraction Logic Flow
The TaskAssistant implements a single-stage analysis using Gemini. It performs vector similarity searches to ensure that extracted tasks are not duplicates of existing active or deleted items. Titles are strictly validated to ensure they are verb-first and contain at least 6 words including specific proper nouns.
Sources: desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/TaskExtraction/TaskAssistant.swift:105-150, desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/TaskExtraction/TaskAssistant.swift:425-450, desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/TaskExtraction/TaskAssistant.swift:650-675
Onboarding & Knowledge Graph
The onboarding process is a multi-step conversational journey managed by the OnboardingPagedIntroCoordinator. It focuses on building a "Knowledge Graph" (KG) of the user's digital life before completing the setup.
Onboarding Steps
- Identity Confirmation: Confirms name and language preferences, seeding the KG with a central "person" node.
- File Scan: Scans
~/Documents,~/Downloads, and~/Developerto identify projects, technologies, and applications. - Permission Requests: Systematically requests Microphone, Notifications, Accessibility, and Screen Recording.
- Insight Generation: Reads local email and calendar data to synthesize a profile and suggest monthly goals.
- Completion: Finalizes the setup via
complete_onboardingtool, enabling background services.
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:450-550, desktop/macos/Desktop/Sources/OnboardingPagedIntroCoordinator.swift:200-250
Knowledge Graph Structure
The KG uses nodes of types person, organization, place, thing, and concept. Edges define relationships such as works_on, uses, prefers, and prioritizes. This graph is visualized live during the onboarding chat.
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:475-485, desktop/macos/Desktop/Sources/OnboardingPagedIntroCoordinator.swift:450-480
Automation & Developer Tools
Omi includes a DesktopAutomationBridge that enables programmatic control of the app in non-production builds. This allows for cursor-free interaction and rapid testing.
CLI Control (omi-ctl)
| Command | Action |
|---|---|
navigate <screen> | Jumps to a specific screen (e.g., rewind, memories). |
state | Returns a JSON snapshot of auth, onboarding, and current tab. |
action <name> | Runs semantic actions like refresh_all_data or toggle_transcription. |
UI Verification (agent-swift)
Developers use agent-swift to interact with the SwiftUI accessibility tree. It supports commands like click (CGEvent for SwiftUI) and press (AXPress for AppKit/Settings).
Sources: desktop/macos/e2e/SKILL.md:25-50, desktop/macos/AGENTS.md:145-165
Conclusion
The macOS app is a sophisticated integration of native system monitoring and agentic AI. By combining proactive task extraction with a deep capability registry, Omi provides a contextual interface that understands the user's work patterns. Its local-first architecture ensures privacy and performance, while its extensive automation bridge allows for robust developer iteration and E2E verification.