Mobile App Conversation Capturing
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Mobile App Conversation Capturing is a core functional layer of the Omi platform, responsible for the real-time acquisition, transcription, and processing of verbal interactions. The system supports two primary input vectors: the mobile device's internal microphone and external Omi wearable hardware connected via Bluetooth Low Energy (BLE). This module serves as the entry point for the "2nd brain" architecture, transforming raw audio into structured data like summaries and action items.
The capturing process is integrated within the Flutter-based mobile application, leveraging native bridges for hardware communication and a Python-based backend for heavy-duty LLM processing and transcription refinement. Users can monitor live transcripts and waveforms during the session, with the system providing background persistence and synchronization.
Sources: README.md:5-10, app/e2e/SKILL.md:144-150, app/AGENTS.md:52-58
System Architecture and Input Vectors
The capturing architecture is split between the Flutter mobile client and the Omi backend. The mobile app handles the UI state, microphone management, and BLE connectivity, while the backend manages the transcription pipeline (STT), voice activity detection (VAD), and speaker diarization.
Data Flow Overview
The following diagram illustrates the flow of audio data from the input source to the processed conversation state.
The diagram shows the transition from raw audio input to structured processing via the CaptureProvider and backend services.
Sources: README.md:100-120, app/AGENTS.md:33-40, app/AGENTS.md:52-58
Key Components
| Component | Responsibility | Source File |
|---|---|---|
CaptureProvider | Manages capturing state, binary WebSocket streaming, and device connection. | app/AGENTS.md:88 |
RecorderHostApi | Pigeon-generated interface for native audio/BLE recording (iOS/Android). | app/AGENTS.md:33-36 |
Capturing Page | Flutter UI displaying live transcript, waveform, and stop button. | app/e2e/SKILL.md:148-150 |
Device Settings | Configures BLE hardware gain, mic gain, and LED brightness. | app/e2e/SKILL.md:183-186 |
Capturing Workflows
Phone Microphone Capture
When using the phone's microphone, the application requests RECORD_AUDIO permissions and initializes the native audio recording bridge. The UI transitions to the Capturing state, showing a live waveform and a real-time transcript produced by the backend's streaming STT pipeline.
Sources: app/e2e/flows/conversation-capturing.yaml:10-25, app/AGENTS.md:53
Wearable Device Capture (BLE)
Capture via Omi hardware involves a specialized BLE handshake. The app scans for the wearable, establishes a connection, and receives audio data over a custom BLE protocol.
Sequence of events for a BLE-based conversation capture session.
Sources: app/e2e/flows/device-capture.yaml, README.md:100-120
Conversation Processing Logic
Once the audio stream is finalized, the backend performs several structured extraction steps. This logic is primarily housed in backend/utils/llm/conversation_processing.py.
Discard Logic
To prevent cluttering the user's "2nd brain" with trivial noise, the system employs an LLM-based discard check. Short snippets (under 2 minutes) are scrutinized for actionable content.
- Keep Criteria: Action items, plans, personal facts, or visually significant scenes.
- Discard Criteria: Trivial snippets, greetings ("hello", "okay"), or fragmented speech.
Sources: backend/utils/llm/conversation_processing.py:348-385
Structured Extraction
The system extracts metadata, titles, and summaries from the finalized transcript. If participant names are available from calendar context, the LLM is instructed to map "Speaker X" labels to real names.
The extraction pipeline converts raw transcript text into a structured Conversation object.
Sources: backend/utils/llm/conversation_processing.py:730-760, backend/utils/llm/conversation_processing.py:465-490
Action Item Extraction Rules
The system follows strict rules for task extraction:
- Clear Ownership: Focus on the primary user's tasks.
- Timing Signals: Must have explicit or relative timing cues.
- Strict Filtering: Excludes vague suggestions, hypothetical scenarios, or routine tasks.
- Vague Reference Resolution: Replaces pronouns like "it" or "this project" with specific nouns from the context.
Sources: backend/utils/llm/conversation_processing.py:535-585
Permissions and Settings
Capturing requires a specific matrix of permissions depending on the input vector and platform.
| Permission | Android Requirement | iOS Requirement | Usage |
|---|---|---|---|
| Microphone | RECORD_AUDIO | NSMicrophoneUsageDescription | Local phone recording |
| Bluetooth | BLUETOOTH_SCAN/CONNECT | NSBluetoothAlwaysUsageDescription | BLE Device communication |
| Background | FOREGROUND_SERVICE | UIBackgroundModes | Continuous capture |
Sources: app/AGENTS.md:52-65
Conclusion
Mobile App Conversation Capturing provides the foundational data for the Omi ecosystem. By combining flexible input vectors (Phone Mic and BLE) with a sophisticated backend processing pipeline, the system ensures that verbal interactions are not just recorded, but transformed into actionable memories. The integration of LLM-based discard logic and name resolution ensures high-quality data retention within the mobile user interface.