Action Items & Goal Tracking
Relevant source files
The following files were used as context for generating this wiki page:
The Action Items & Goal Tracking system within the Omi project is a core intelligence layer designed to identify, extract, and manage user commitments and long-term objectives. It leverages Large Language Models (LLMs) to analyze multi-modal inputs—including conversation transcripts, photos from wearable devices, and desktop screen activity—to transform raw data into structured, actionable tasks and measurable goals.
This system spans both the backend infrastructure, where extraction logic and database interactions reside, and the desktop/mobile clients, which provide user interfaces for management and proactive assistance. By integrating these components, Omi serves as a "2nd brain," helping users maintain agency over their daily commitments and long-term achievements. Sources: README.md, backend/utils/llm/conversation_processing.py:342-358, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:186-200
Action Item Extraction Architecture
Action items are primarily extracted through two pathways: Conversation Processing (from audio/wearable data) and Proactive Desktop Assistance (from screen context).
1. Conversation-Based Extraction
The backend processes transcripts and photos to identify future-facing commitments. The extraction logic applies aggressive filtering to ensure quality, prioritizing tasks with clear ownership, concrete actions, and timing signals. Sources: backend/utils/llm/conversation_processing.py:381-428
The diagram above illustrates the logical flow of extracting action items from conversation content.
Sources: backend/utils/llm/conversation_processing.py:461-550
2. Proactive Task Extraction (macOS)
On desktop, the TaskAssistant analyzes screenshots to detect commitments. It uses a whitelist of allowed applications (e.g., Slack, Telegram, Gmail) and specific browser keywords to focus on relevant context. Sources: desktop/macos/Desktop/Sources/ProactiveAssistants/Assistants/TaskExtraction/TaskAssistantSettings.swift:18-63
| Feature | Description |
|---|---|
| Whitelisting | Only extracts from specific productivity and messaging apps. |
| Commitment Detection | Identifies patterns like "I'll handle it" or "Got it". |
| Unaddressed Requests | Detects when others ask the user for action without a response. |
| Confidence Threshold | Uses a default minimum confidence of 0.75 for extraction. |
Data Models
Action Item Schema
The system uses a structured model for tasks, including metadata about their origin and priority. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:513-547
| Field | Type | Description |
|---|---|---|
description | String | Timeless, specific description (e.g., "Buy groceries"). |
due_at | DateTime | Resolved local ISO 8601 timestamp, converted to UTC. |
priority | Enum | high | medium | low. |
source | String | screenshot | conversation | omi | manual. |
completed | Boolean | Whether the task is finished. |
agent_status | Enum | pending | processing | editing | completed | failed. |
Sources: backend/utils/llm/conversation_processing.py:580-605, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:515-542
Goal Schema
Goals represent higher-level objectives that can be linked to individual tasks. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:604-617
| Field | Type | Description |
|---|---|---|
title | String | Short name shown in UI. |
goal_type | Enum | boolean | scale | numeric. |
target_value | Numeric | Target value for completion. |
current_value | Numeric | Current progress toward the goal. |
Sources: backend/database/goals.py, desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:605-612
Extraction Logic & Filtering
The extraction process follows strict rules to avoid "garbage" tasks:
- Deduplication: LLMs are instructed to be conservative, only suppressing candidates if they are 100% confident a duplicate exists. Sources: backend/utils/llm/conversation_processing.py:466-485
- Time Normalization: All time references (e.g., "tomorrow") are removed from the description and placed in the
due_atfield. Dates are resolved based on the user's local timezone. Sources: backend/utils/llm/conversation_processing.py:559-575 - Exclusion Rules: Casual suggestions ("we should grab coffee"), hypothetical scenarios, and tasks already in progress are explicitly excluded. Sources: backend/utils/llm/conversation_processing.py:535-555
# Example of date normalization logic
def _normalize_action_item_due_dates(action_items, *, user_tz, now, log_past_due_clears):
for action_item in action_items:
if action_item.due_at is None:
continue
# Convert to UTC and handle timezone math
action_item.due_at = action_item.due_at.astimezone(timezone.utc)
# Clear past due items (> 1 day old)
if action_item.due_at < now - timedelta(days=1):
action_item.due_at = None
return action_items
Sources: backend/utils/llm/conversation_processing.py:91-109
AI Mentor Integration
Goals and Action Items are central to the AI Mentor chat experience. The Omi mentor acts as a "smart friend" who provides feedback based on the user's current goals and history. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:186-193
Mentor Behavior Guidelines
- Directness: The AI calls out gaps between a user's plan and their goal directly. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:203-207
- Conciseness: Responses are typically 2-8 lines, avoiding summaries of what the user just said. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:211-218
- Personalization: Retreives
user_goalsandaction_itemsto inform advice. Sources: backend/utils/llm/chat.py:382-398
Sequence illustrating how goals and tasks are retrieved to provide personalized AI mentoring.
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:426-455, backend/utils/llm/chat.py:408-418
Conclusion
The Action Items & Goal Tracking system is the functional backbone of Omi's proactive utility. By combining cross-platform capture (desktop and wearable) with sophisticated LLM-based extraction and an AI Mentor interface, it ensures that transient conversational commitments and long-term intentions are captured, organized, and acted upon. Sources: README.md, backend/utils/llm/conversation_processing.py:651-660