Voice Activity Detection & Diarization
Relevant source files
The following files were used as context for generating this wiki page:
The Voice Activity Detection (VAD) and Diarization systems within Omi are critical components of the real-time transcription pipeline. These systems work together to distinguish human speech from background noise and identify individual speakers within a conversation. This ensures that the AI assistant, Omi, can accurately attribute statements to specific participants and generate meaningful summaries, memories, and action items.
The VAD component acts as a gatekeeper, identifying segments of audio that contain speech to be processed by Speech-to-Text (STT) services like Deepgram or Parakeet. Diarization then analyzes these speech segments to produce speaker embeddings and identification, allowing the system to label "Speaker 0", "Speaker 1", etc., or match them to known individuals in the user's contacts.
Sources: README.md:95-115, AGENTS.md:120-140
System Architecture
The VAD and Diarization logic is distributed across specialized microservices and utility modules. The backend service orchestrates calls to these components during the conversation lifecycle.
The diagram shows the flow of audio data from various devices through the VAD, Diarizer, and STT services to generate enriched transcripts.
Sources: README.md:95-115, AGENTS.md:120-140
Key Components
| Component | Responsibility | Technical Stack |
|---|---|---|
| VAD Service | Detects voice activity and identifies speaker boundaries. | Modal / GPU |
| Diarizer Service | Generates speaker embeddings at /v2/embedding. | Python / GPU |
| Speaker Identification | Matches embeddings against user-specific profiles. | Python Utilities |
| Transcription Sync | Coordinates VAD, STT, and diarization for local files. | Cloud Run |
Sources: AGENTS.md:120-145, backend/utils/llm/conversation_processing.py:100-110
Transcription Pipeline Integration
The VAD and Diarization results are integrated into the final transcript via the conversation_processing utility. This module handles the logic for formatting transcripts and attributing text to specific speakers.
Speaker Attribution Logic
The system uses speaker IDs (e.g., Speaker 0, Speaker 1) when specific names are not identified. However, if calendar meeting context is available, the system attempts to match these IDs to actual participant names.
- Speaker Mapping: The system analyzes transcript context against
CalendarMeetingContext(meeting title, participants, and notes) to replace generic labels with real names. - Segment Formatting: Transcripts are stored as
TranscriptSegmentobjects containing the text, speaker index, and timing information.
Sources: backend/utils/llm/conversation_processing.py:465-485, backend/utils/llm/chat.py:530-550
This sequence illustrates the parallel coordination required to produce a diarized transcript.
Sources: AGENTS.md:130-150, backend/utils/llm/conversation_processing.py:450-460
Speaker Identification Features
Speaker identification extends basic diarization by attempting to recognize specific individuals across different conversations.
Memory & Contextual Recognition
The system uses the memories database and people metadata to improve identification:
- Speaker Embeddings: Generated for each unique voice detected in a session.
- Person Matching: The
retrieve_memory_context_paramsfunction usesperson_idsto fetch known people data from the database. - Prompt Enrichment: When generating summaries or chat responses, the system injects known speaker labels into the LLM context to provide personalized responses (e.g., "You talked to Sarah about...").
Sources: backend/utils/llm/chat.py:480-500, backend/utils/llm/conversation_processing.py:500-520
Technical Implementation Details
# From backend/utils/llm/conversation_processing.py
def _local_started_at_iso(started_at: datetime, tz: Optional[str]) -> str:
"""
Renders capture time as user's local wall-clock for prompt date context.
Prevents mislabeling of time of day in diarized summaries.
"""
try:
user_tz = ZoneInfo(tz) if tz else timezone.utc
except Exception:
user_tz = timezone.utc
aware = started_at if started_at.tzinfo is not None else started_at.replace(tzinfo=timezone.utc)
return aware.astimezone(user_tz).replace(tzinfo=None).isoformat()
Sources: backend/utils/llm/conversation_processing.py:596-608
Conclusion
The Voice Activity Detection and Diarization systems are fundamental to Omi's ability to act as a "second brain." By accurately segmenting audio and identifying speakers, Omi can provide context-aware insights, attribute action items to the correct individuals, and maintain a detailed, multi-participant memory of the user's interactions. The integration of calendar context further bridges the gap between raw audio data and high-level social understanding.
Sources: README.md:15-25, backend/utils/llm/conversation_processing.py:470-490