Real-time Transcription System
Relevant source files
The following files were used as context for generating this wiki page:
The Real-time Transcription System in Omi is a multi-layered pipeline designed to capture, process, and analyze audio data from wearables and desktop applications as it happens. The system utilizes a combination of Voice Activity Detection (VAD), speaker diarization, and Speech-to-Text (STT) services to transform raw audio into structured, actionable conversational data.
This system acts as the "ear" of the Omi ecosystem, feeding transcribed text into the larger "2nd brain" architecture for memory ingestion, proactive insights, and task extraction. Sources: README.md:3, AGENTS.md:123-145
System Architecture and Data Flow
The transcription pipeline follows a specific path from hardware capture to structured memory. Audio is streamed from devices (Omi Wearable, macOS App, or Mobile App) via BLE, HTTPS, or WebSockets to the backend. Sources: README.md:65-80
Processing Components
The backend architecture is divided into several specialized services:
| Component | Function | Technology |
|---|---|---|
| Pusher | Receives binary audio via WebSockets and handles speaker sample extraction. | Python/WebSockets |
| VAD | Voice Activity Detection to identify when speech is occurring. | GPU (ONNX) |
| Diarizer | Generates speaker embeddings to identify unique participants. | GPU |
| STT Engine | Transcribes audio to text using Deepgram (self-hosted or cloud) or Parakeet. | Deepgram / Parakeet |
| Processing | Segments transcripts, extracts action items, and identifies calendar events. | LLMs (GPT-4o/Claude) |
Sources: AGENTS.md:131-150, backend/README.md:45-55
High-Level Data Flow
The following diagram illustrates how audio moves through the transcription and processing layers:
The system uses an asynchronous supervisor loop for WebSockets to ensure concurrent handling of audio streams and background tasks. Sources: AGENTS.md:112-118, README.md:65-80
Conversation Structuring and Filtering
Once a transcript is generated, it undergoes several refinement steps to ensure high-quality data ingestion. The system evaluates whether a conversation is "memory-worthy" based on duration and content complexity. Sources: backend/utils/llm/conversation_processing.py:400-430
Discard Logic
Conversations are discarded if they are trivial (e.g., brief greetings, filler words, or fragmented speech). A conversation is kept if it contains:
- Action items or commitments.
- Personal facts or preferences.
- Visual context from wearable cameras (photos).
- Significant social interactions.
Sources: backend/utils/llm/conversation_processing.py:435-460
Structured Extraction
The system transforms raw transcripts into a Structured data model, extracting:
- Title: A headline capturing the central topic (e.g., "Team Finalizes Budget").
- Overview: A condensed summary of main points.
- Category: Classification (Work, Personal, etc.).
- Action Items: Specific tasks for the user.
- Events: Confirmed commitments or meetings.
Sources: backend/utils/llm/conversation_processing.py:734-755
Speaker Identification and Diarization
The system handles multiple participants by mapping speakers identified by the Diarizer to real-world identities. Sources: backend/utils/llm/conversation_processing.py:530-545
- Calendar Integration: If a transcription session coincides with a calendar event, the system uses participant names from the meeting context to replace generic "Speaker 0" or "Speaker 1" labels. Sources: backend/utils/llm/conversation_processing.py:516-529
- Speaker Mapping: The Diarizer service provides speaker embeddings, which are matched against known user profiles to identify the primary user and their frequent contacts. Sources: AGENTS.md:143-145
Technical Implementation Details
Action Item Extraction Logic
The extraction of tasks is governed by strict rules to prevent "flooding" the user with trivial data. The system prioritizes ownership, concrete action, and timing signals. Sources: backend/utils/llm/conversation_processing.py:560-600
# Example of timestamp resolution for action items
# Sources: [backend/utils/llm/conversation_processing.py:695-710]
def _normalize_action_item_due_dates(action_items, user_tz, now):
for action_item in action_items:
if action_item.due_at:
# Convert naive local time to UTC
action_item.due_at = action_item.due_at.replace(tzinfo=user_tz).astimezone(timezone.utc)
# Discard if more than 1 day in the past
if action_item.due_at < now - timedelta(days=1):
action_item.due_at = None
Database Schema for Transcription
The system maintains the following tables in the local desktop database to support real-time features:
| Table | Column | Description |
|---|---|---|
transcription_sessions | startedAt, finishedAt, status, overview | Metadata for the entire recording session. |
transcription_segments | sessionId, speaker, text, startTime | Individual lines of the transcript with speaker ID. |
live_notes | sessionId, text, isAiGenerated | AI-generated notes created during a live recording. |
Sources: desktop/macos/Desktop/Sources/Chat/ChatPrompts.swift:398-435
Conclusion
The Real-time Transcription System serves as the foundation for Omi's ability to act as a proactive assistant. By combining low-level audio processing with high-level LLM-driven structured extraction, it converts ambient audio into a durable knowledge graph that the user can query and utilize for task management. Sources: README.md:3-8, backend/utils/llm/conversation_processing.py:1005-1015