Monitoring, Logging & Alerts
Relevant source files
The following files were used as context for generating this wiki page:
- backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/general.rules.yaml
- backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml
- backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/charts/crds/crds/crd-alertmanagerconfigs.yaml
- AGENTS.md
- desktop/macos/AGENTS.md
- web/admin/app/api/omi/releases/route.ts
- backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/alertmanager/servicemonitor.yaml
- plugins/apps-js/README.md
The Omi ecosystem utilizes a multi-layered observability strategy encompassing infrastructure monitoring, application-level logging, and automated alerting. This system ensures high availability for backend services, tracks the health of desktop and mobile releases, and maintains security by sanitizing sensitive data before it reaches logs. The monitoring stack is primarily built on the kube-prometheus-stack, while application metrics are aggregated via PostHog and Sentry.
Infrastructure Monitoring
The core infrastructure monitoring is powered by Prometheus and Grafana, deployed via Helm charts. It tracks cluster-level resources, including CPU, memory, and disk utilization, specifically for various backend components like the Deepgram self-hosted STT service.
Prometheus Configuration
Prometheus is configured as a Custom Resource (CR) that orchestrates scraping intervals, evaluation intervals, and retention policies. It targets various services through ServiceMonitor resources, which define the endpoints and paths (typically /metrics) where telemetry data is exposed.
The diagram shows the relationship between the Prometheus Operator, the Prometheus instance, and the various scrape targets within the cluster.
Sources: backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml, backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/alertmanager/servicemonitor.yaml:30-45
Alerting Pipeline
Alerts are defined using PrometheusRule resources. A critical "Watchdog" alert is kept constantly firing to certify that the entire alerting pipeline (Prometheus to Alertmanager to Receiver) is functional.
| Alert Name | Severity | Description |
|---|---|---|
| TargetDown | Warning | Fired when >10% of targets in a namespace are unreachable. |
| Watchdog | None | Always firing to verify the alerting pipeline is alive. |
| InfoInhibitor | None | Used to inhibit noisy info-level alerts when warning/critical alerts are active. |
Application Logging & Security
The project enforces strict logging security protocols to prevent the exposure of Personal Identifiable Information (PII) and sensitive data.
Log Sanitization
Backend services must use the sanitize() and sanitize_pii() utilities from utils.log_sanitizer.
sanitize(): Applied toresponse.text, API responses, and error bodies.sanitize_pii(): Applied to names, emails, and user-generated text.- Exceptions: UIDs, IPs, and HTTP status codes are kept visible to facilitate debugging.
Sources: AGENTS.md:66-72
release Health Monitoring
Desktop and web release health is tracked using a combination of Sentry and PostHog.
- Sentry: Used for tracking crashes and exceptions in the macOS desktop app. Developers use scripts like
sentry-release.shto analyze issues in specific versions. - PostHog: Utilized for aggregating behavioral metrics and system failures (e.g., "App Crash Detected", "Screen Capture Broken Detected").
Sources: desktop/macos/AGENTS.md:12-20, web/admin/app/api/omi/releases/route.ts:75-95
Release Metrics & Analytics
The admin dashboard aggregates release-specific metrics to calculate a "Release Rating." This rating is derived from crash rates and user feedback.
Metric Calculation Logic
The system calculates metrics over a rolling 60-day window using PostHog's HogQL.
Sequence of events showing how the admin dashboard aggregates cross-platform data to evaluate release health.
Sources: web/admin/app/api/omi/releases/route.ts:10-50, web/admin/app/api/omi/releases/route.ts:115-160
Performance Metrics Table
| Metric | Event Source | Description |
|---|---|---|
| Crash Rate | App Crash Detected / App Launched | Ratio of crashes per launch. >5% triggers a "high crash rate" warning. |
| Broken Count | Screen Capture Broken Detected | Tracks failures in the desktop screen capture module. |
| User Feedback | Feedback Submitted | Direct user complaints submitted via the app interface. |
Sources: web/admin/app/api/omi/releases/route.ts:117-150
Integration Monitoring
Plugins and external integrations (e.g., Node.js based "OmiSend") include built-in performance monitoring for webhooks and API interactions.
Monitored Plugin Statistics
- Webhook Processing: Statistics on processing time and success rates for voice segments.
- Email Metrics: Tracking of AI-powered email composition and sending success.
- System Health: Standard
/healthendpoints for connectivity checks.
Sources: plugins/apps-js/README.md:140-150
Summary
The Omi monitoring system provides a comprehensive view from low-level Kubernetes resources to high-level user experience metrics. By leveraging Prometheus for infrastructure, strict sanitization for logging, and PostHog/Sentry for release health, the architecture ensures that developers can respond to infrastructure failures and application regressions while maintaining user privacy.