High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
Omi is an open-source AI ecosystem designed to act as a "second brain" by capturing screen context and real-time conversations. The architecture spans across specialized hardware wearables, cross-platform client applications (macOS, iOS, Android), and a Python-based backend that orchestrates transcription, Voice Activity Detection (VAD), diarization, and Large Language Model (LLM) processing. Sources: README.md:3-8
System Overview
The Omi ecosystem is divided into three primary tiers: the device layer (hardware), the client layer (desktop and mobile apps), and the services layer (backend and AI infrastructure).
Core Components
| Component | Description | Tech Stack |
|---|---|---|
| Wearables | Continuous audio capture and BLE streaming devices. | nRF5340, Zephyr RTOS, C |
| macOS App | Desktop client for screen capture and local intelligence. | Swift, SwiftUI, Rust |
| Mobile App | Interface for on-the-go conversation management. | Flutter (iOS & Android) |
| Backend API | Central hub for data persistence and AI orchestration. | Python, FastAPI, Firebase |
| AI Services | Specialized modules for STT, VAD, and Diarization. | Deepgram, Modal (GPU), LLM Gateway |
Sources: README.md:65-85, AGENTS.md:144-155, omi/hardware/consumer/README.md:46-56
Architectural Flow Diagram
The following diagram illustrates the relationship between user devices and the centralized backend services.
Sources: README.md:45-63
Backend Architecture
The backend is a distributed system primarily built with FastAPI. It handles high-concurrency audio streaming and complex LLM processing pipelines.
Service Map
The backend is decomposed into several specialized services to handle GPU-intensive tasks and real-time streaming:
- Main Backend (main.py): The entry point for REST APIs. It handles the orchestration of audio streams to the "pusher" via WebSockets. Sources: AGENTS.md:144-147
- Pusher (pusher/main.py): Receives binary audio data over WebSockets. It calls the diarizer and Deepgram for speaker identification and sample extraction. Sources: AGENTS.md:162-164
- LLM Gateway (llm_gateway/main.py): An internal service for Omi-managed LLM lanes, such as
omi:auto:*routes, used for processing chat completions without direct client exposure. Sources: AGENTS.md:160-161 - Diarizer & VAD: GPU-accelerated services (often hosted on Modal) that provide speaker embeddings and voice activity detection. Sources: AGENTS.md:167-170
- Backend-Sync: A Cloud Run service utilized for long-running jobs like audio merging and local file synchronization via Cloud Tasks. Sources: AGENTS.md:173-178
Async Concurrency and Blocking Calls
The architecture mandates a strict separation between synchronous and asynchronous code to prevent event loop blocking. Sync SDKs (like Firestore and GCS) must be wrapped in run_blocking calls using specialized executors:
critical_executor: Auth and rate-limiting.db_executor: Firestore and Redis CRUD operations.llm_executor: LLM API calls.storage_executor: GCS and file I/O.
Sources: AGENTS.md:85-93
Client Architecture
macOS Desktop Client
The macOS application follows a hybrid architecture combining a Swift/SwiftUI frontend for the native user interface and a Rust-based backend for performance-critical local processing. Sources: desktop/macos/AGENTS.md:7-10
- Local Intelligence: Uses a local SQLite database to store captured screen frames (OCR text), conversation segments, and knowledge graph nodes. Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:458-475
- Automation Bridge: Includes a local automation bridge (running on port 47777 by default) that allows in-process semantic actions and state inspection. Sources: AGENTS.md:286-300
Mobile (Flutter) Client
The mobile app acts as the primary gateway for the Omi wearable device.
- Native Bridge (Pigeon): Uses a bidirectional bridge between Flutter and native iOS/Android code for recording, audio management, and BLE communication. Sources: app/AGENTS.md:25-30
- State Management: Utilizes providers like
capture_provideranddevice_providerto manage the lifecycle of real-time audio capture. Sources: app/AGENTS.md:65-67
Data Pipeline: Conversation Processing
Omi employs a multi-stage pipeline for transforming raw audio into structured intelligence.
Structured Extraction
The system uses LLMs to perform several extraction tasks on transcripts:
- Discard Logic: Determines if a conversation is meaningful enough to save. Sources: backend/utils/llm/conversation_processing.py:461-465
- Action Item Extraction: Identifies concrete tasks, ownership, and due dates. Sources: backend/utils/llm/conversation_processing.py:596-605
- Transcript Structuring: Generates titles, evocative emojis, and summaries. Sources: backend/utils/llm/conversation_processing.py:848-855
Knowledge Graph Synthesis
Clients (specifically desktop) synthesize a local Knowledge Graph by querying local files and memories. This involves:
- Node Types: person, organization, place, thing, concept.
- Relationships: "works on", "uses", "collaborates with".
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:537-545
Hardware Architecture: Omi Consumer
The Omi wearable is built on an open-source hardware design centered around Nordic Semiconductor components. Sources: omi/hardware/consumer/README.md:1-5
| Feature | Specification |
|---|---|
| Processor | nRF5340 dual-core Bluetooth LE SoC |
| Wi-Fi | nRF7002 Wi-Fi 6 chip |
| Microphones | 2x TDK T5838 top-port PDM microphones |
| Storage | 4Gbit NAND Flash (CSNP4GCR01) |
| Power | 3.7V 150mAh LiPo with magnetic pogo pin charging |
Sources: omi/hardware/consumer/README.md:46-56
Conclusion
Omi's high-level architecture is a sophisticated blend of real-time audio processing and AI-driven personal intelligence. By distributing tasks between local desktop environments, mobile native bridges, and a GPU-accelerated Python backend, the system achieves a balance of privacy (local storage) and power (cloud-based LLM extraction). This structure supports continuous, ambient capture across multiple device types while maintaining a unified data model centered on user "memories" and actionable insights. Sources: README.md:10-15, AGENTS.md:144-150