Kubernetes & Helm Deployment

Relevant source files

The following files were used as context for generating this wiki page:

• backend/charts/backend-listen/templates/deployment.yaml
• backend/charts/agent-proxy/templates/deployment.yaml
• backend/charts/pusher/templates/deployment.yaml
• backend/charts/vad/templates/deployment.yaml
• backend/charts/parakeet/templates/deployment.yaml
• backend/charts/deepgram-self-hosted/nova-3/templates/api/api.deployment.yaml
• AGENTS.md
• backend/charts/monitoring/

Introduction

The Omi project utilizes a microservices architecture deployed on Kubernetes (specifically Google Kubernetes Engine - GKE) managed via Helm charts. The deployment strategy encompasses a variety of internal services, including audio processing, voice activity detection (VAD), speaker identification, and agent proxies, all coordinated through a centralized backend system.

The infrastructure is designed for high availability and scalability, employing Kubernetes features such as Horizontal Pod Autoscaling (HPA), Prometheus-based monitoring, and secure secret management via ExternalSecrets. The deployment also covers self-hosted speech-to-text (STT) solutions like Deepgram and Parakeet, alongside specialized components like the agent-vm-reaper for managing transient compute resources.
Sources: AGENTS.md:128-150, backend/charts/monitoring/

Deployment Architecture

The system is divided into several discrete Helm charts located in the backend/charts/ directory. Each service is deployed as a standard Kubernetes Deployment or StatefulSet with associated services for internal and external communication.

Core Service Components

The following diagram illustrates the high-level relationship between the primary deployed services:

Key Deployment Modules

Module	Path	Description
backend-listen	backend/charts/backend-listen/	The main REST API entry point for the backend.
pusher	backend/charts/pusher/	Handles binary WebSocket audio streams for processing.
agent-proxy	backend/charts/agent-proxy/	Proxies WebSocket connections to User Agent VMs.
vad	backend/charts/vad/	GPU-accelerated Voice Activity Detection service.
monitoring	backend/charts/monitoring/	Prometheus, Grafana, and Loki stack for observability.
deepgram-self-hosted	backend/charts/deepgram-self-hosted/	Self-hosted STT engine and API.
Sources: AGENTS.md:144-150

Service Configuration & Environment

Deployment manifests utilize standard Helm templating for managing environment variables, resource constraints, and probes.

Common Deployment Structure

Most services follow a consistent deployment pattern including:

• Probes: Liveness, Readiness, and Startup probes typically targeting port 8080 or the service's primary port.
• Resource Management: Specific CPU and Memory requests/limits defined in values.yaml.
• Secret Injection: Environment variables are often sourced from secretRef or valueFrom configurations, particularly for sensitive credentials like GOOGLE_CLOUD_PROJECT or DEEPGRAM_API_KEY.

Sources: backend/charts/backend-listen/templates/deployment.yaml:65-103, backend/charts/vad/templates/deployment.yaml:48-84

GPU and specialized resources

Services like vad and deepgram-engine require GPU resources. Deployment manifests explicitly request nvidia.com/gpu under container resources to ensure scheduling on appropriate node pools.
Sources: backend/charts/deepgram-self-hosted/nova-2/templates/engine/engine.deployment.yaml:70-80, backend/charts/vad/dev_omi_vad_values.yaml:126-130

Monitoring and Observability

The monitoring chart integrates the kube-prometheus-stack to provide comprehensive cluster visibility.

Prometheus Rules

Alerts are defined using PrometheusRule objects to monitor cluster health, including API server errors and pod status.

• KubeClientErrors: Alerts when the API server experiences high error rates (>1%).
• KubePodCrashLooping: Triggers when pods enter CrashLoopBackOff state.
• KubeVersionMismatch: Detects inconsistent Kubernetes component versions across the cluster.

Sources: backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system.yaml:61-75, backend/charts/deepgram-self-hosted/nova-3/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-apps.yaml:26-40

Monitoring Flow

Deployment Workflow

The project enforces a strict deployment contract between Helm values and runtime environments.

Deployment Commands

• Backend Deployment: Executed via GitHub workflow: gh workflow run gcp_backend.yml -f environment=prod -f branch=main.
• Validation: Post-deployment validation is performed using python backend/scripts/validate-backend-runtime-env.py --env <dev|prod>.
• Secret Syncing: backend-secrets manages ExternalSecret and SecretStore resources to sync runtime secrets from cloud providers into GKE namespaces.

Sources: AGENTS.md:152-155, AGENTS.md:275-276, AGENTS.md:150

Conclusion

The Omi Kubernetes & Helm deployment strategy provides a robust foundation for scaling AI-driven audio processing services. By leveraging modular Helm charts and deep integration with Prometheus monitoring, the architecture ensures operational stability while maintaining the flexibility to run complex GPU-dependent workloads alongside standard API services.