Testing Strategies & E2E Flows
Relevant source files
The following files were used as context for generating this wiki page:
Omi employs a multi-layered testing strategy designed to verify functionality across its mobile (Flutter) and desktop (macOS) applications. The strategy prioritizes End-to-End (E2E) flows that simulate real user journeys, supported by specialized automation tools like agent-flutter and agent-swift. These tools interact with live app instances through specialized debug protocols and Accessibility APIs to ensure that code changes maintain high-quality user experiences.
The core of the E2E infrastructure is the Flow Walker pipeline, which uses YAML-based flow definitions to execute, verify, and report on complex cross-screen interactions. This system is designed to provide high-fidelity evidence of feature correctness, including screenshots and video recordings, while maintaining a clear mapping between source code and the user-facing features they support.
Sources: app/e2e/FLOW-WALKER-SKILL.md:1-15, desktop/macos/e2e/SKILL.md:1-10, AGENTS.md:310-325
E2E Testing Infrastructure
The E2E testing architecture varies by platform but shares a common philosophy of driving live app instances to verify state and UI correctness.
Platform Automation Tools
| Tool | Platform | Mechanism | Key Command |
|---|---|---|---|
agent-flutter | Mobile (Flutter) | Marionette debug protocol | agent-flutter snapshot -i |
agent-swift | Desktop (macOS) | macOS Accessibility API | agent-swift snapshot -i |
omi-ctl | Desktop (macOS) | Local HTTP automation bridge | ./scripts/omi-ctl navigate <screen> |
Sources: app/e2e/SKILL.md:15-30, desktop/macos/e2e/SKILL.md:70-80, desktop/macos/e2e/SKILL.md:40-50
Flow Walker Pipeline
The Flow Walker is an agent-first testing CLI used to manage the E2E lifecycle. It translates YAML flow definitions into executable steps.
The diagram shows the 6-stage pipeline from initialization to cloud reporting. Sources: app/e2e/FLOW-WALKER-SKILL.md:65-115
Mobile (Flutter) Testing Strategy
Mobile testing focuses on widget-level interactions and state management verification.
Test Structure
- Unit Tests: Located in
test/unit/, covering Auth, tokens, preferences, and audio utilities. - Widget Tests: Located in
test/widgets/, testing UI components like waveforms and transcripts. - Provider Tests: Located in
test/providers/, verifying state management logic incapture_provideranddevice_provider.
Sources: app/AGENTS.md:68-75
Automation Logic
Mobile automation utilizes agent-flutter (Marionette) to interact with the widget tree. Because Flutter rebuilds the widget tree aggressively, the system requires frequent snapshots to avoid stale references.
Sources: app/e2e/SKILL.md:44-65, AGENTS.md:210-225
Desktop (macOS) Testing Strategy
Desktop testing utilizes a "Local HTTP Control Bridge" (DesktopAutomationBridge.swift) which is auto-enabled on non-production builds to facilitate fast navigation and semantic actions without using synthetic mouse events.
Fast-Path Iteration
To avoid slow cycles (like manual login and clicking through sidebars), Omi uses several seeding and control scripts:
- Auth Seeding:
omi-auth-dump.shcaptures a session from the dev app to replay into test bundles. - Settings Seeding:
omi-settings-seed.shcopies developer shortcuts and settings. - Direct Navigation:
omi-ctl navigate <target>allows jumping to specific screens (e.g., Rewind, Settings) in ~150ms.
Sources: desktop/macos/e2e/SKILL.md:15-55, desktop/macos/AGENTS.md:140-155
Feature Vector and Priority
The desktop app uses a scoring model to prioritize E2E coverage based on mission criticality and session frequency.
| Category | Priority | Walker Score | Examples |
|---|---|---|---|
| Core Daily | 9-15 | 2-3 | Dashboard, Chat, Sidebar, Audio Recording |
| Core Weekly | 6-12 | 1-2 | Memory Graph, Apps Marketplace, Goals CRUD |
| System | 3-5 | 0-3 | Settings, Auth, Onboarding |
Sources: desktop/macos/e2e/feature-vector.md:5-40
E2E Flow Definitions
Flows are defined in YAML (v2 format) and include metadata regarding the source files they cover.
YAML Schema Example
version: 2
name: tasks-flow
description: browse and create tasks
app: com.omi.computer-macos
covers:
- desktop/Desktop/Sources/MainWindow/Pages/TasksPage.swift
preconditions:
- auth_ready
steps:
- id: S1
name: Navigate
do: "Click Tasks in sidebar"
expect:
- kind: interactive_count
min: 5
Sources: desktop/macos/e2e/flows/tasks.yaml:1-25, app/e2e/FLOW-WALKER-SKILL.md:23-45
Verified Reference Flows
The following flows are established user journeys used for regression testing:
- Login/Logout: Verifies auth state transitions and UI cleanup.
- Onboarding: A multi-step wizard (11 steps on mobile) covering permissions, language, and initial setup.
- Capture: Verifies microphone access and live transcription processing.
- Memories/Tasks: Verifies retrieval, filtering, and CRUD operations for extracted intelligence.
Sources: app/e2e/FLOW-WALKER-SKILL.md:195-210, app/e2e/SKILL.md:90-120, desktop/macos/e2e/SKILL.md:130-150
Verification and Evidence Collection
The final stage of any testing strategy is the generation of evidence to confirm changes.
- Snapshots: Capturing the interactive element tree as JSON for automated verification.
- Screenshots: Capturing the actual visual state via
agent-swift screenshotoradb exec-out screencap. - Audit Reports: Generating HTML reports that combine step descriptions, outcomes, and screenshots into a single timeline.
- Video Generation: Using
ffmpegto stitch sequential E2E screenshots into a reviewable video.
Sources: app/e2e/FLOW-WALKER-SKILL.md:118-135, desktop/macos/e2e/SKILL.md:165-175
Summary
Testing at Omi is a structured process that moves from localized unit/widget tests to complex E2E journeys. By utilizing platform-specific automation agents (agent-flutter and agent-swift) and a standardized Flow Walker pipeline, developers can verify that feature logic and UI state remain consistent across mobile and desktop environments. The use of named test bundles and automation bridges ensures that these E2E flows can be executed quickly during local development without interfering with production installations.