Testing Strategy & QA
Relevant source files
The following files were used as context for generating this wiki page:
The testing strategy for the J.A.R.V.I.S. project is built on the principle of "Testing as a habit of proof." It focuses on validating architectural invariants, contract integrity across service boundaries, and security-critical execution paths. The project utilizes bun:test as its primary testing framework, covering shared packages, backend services, and frontend shell layouts to ensure a stable "Personal AI Operating System" experience.
QA efforts are integrated into the development lifecycle, requiring developers to build and test locally before pushing code. This strategy prioritizes testing behaviors that would impact the end-user rather than merely snapshotting current data states. It exercises real paths for resolvers, identity boundaries, and local/remote adapters to maintain compatibility and performance under realistic loads.
Sources: instructions/AGENTS.md:144-150, CONTRIBUTING.md:46-51
Core Testing Layers
The project structure dictates a multi-layered testing approach, spanning from low-level data contracts to high-level UI invariants.
1. Shared Contract Validation
The packages/shared suite ensures that the communication between the Electron renderer and the Bun backend remains consistent. Tests validate orb states, API error formats, chat message structures, and memory item categories. These tests act as a schema enforcement layer to prevent breaking changes when data structures evolve.
Sources: packages/shared/tests/contracts.test.ts:33-210
2. Backend Service Logic
Backend testing focuses on the loopback-only request handler and the action engine. Key areas include:
- Service Lifecycle: Verification of loopback-only binding, port fallback mechanisms, and health probe responses.
- Feature Endpoints: CRUD operations for memory, pairing code exchanges, and voice status toggling.
- Event Streaming: Validation of Server-Sent Events (SSE) for real-time updates like
service.connectedororb.state.changed.
Sources: backend/tests/service.test.ts:40-192
3. Action Engine & Security
Given the human-in-the-loop architecture, the ActionEngine tests verify that sensitive system actions (shutdown, folder opening, app closing) remain in a proposed state until explicit user approval. They also enforce allowlist safety by rejecting misclassified apps or arbitrary folder paths.
Sources: backend/tests/action-engine.test.ts:4-90
4. Shell Layout Invariants
The frontend testing strategy focuses on "Hermes-style" invariants. Instead of testing visual pixels, it checks the DOM structure and source code for required elements (settings panel, activity feed, composer) and correct positioning (command dock below the stage).
Sources: tests/shell-layout.test.ts:14-49
Testing Architecture
The following diagram illustrates the relationship between the different test suites and the system components they validate.
This flowchart shows how the contract tests underpin the service, while backend tests focus on logic/execution and frontend tests verify the integrity of the UI shell.
Sources: packages/shared/tests/contracts.test.ts, backend/tests/service.test.ts, tests/shell-layout.test.ts
Key Testing Metrics and Validators
The following table summarizes the critical validation functions used in the shared contract test suite.
| Validator Function | Target Data Structure | Key Constraint Validated |
|---|---|---|
isJarvisHealthSnapshot | Health Response | Must contain uptimeSeconds, orbState, and eventStream. |
isJarvisChatMessage | Chat Payload | Role must be 'user' or 'assistant'; content cannot be empty. |
isJarvisModelReadiness | Model Status | Requires instruction if status is model-missing or unreachable. |
isJarvisMemoryItem | Memory Data | Must include a valid category and provenance. |
isJarvisActionIntent | Action Intent | Tracks status transitions (e.g., proposed, completed). |
| Sources: packages/shared/tests/contracts.test.ts:16-160 |
Execution Lifecycle Testing
The backend/tests/service.test.ts file exercises the end-to-end lifecycle of a local service request.
The diagram represents the lifecycle of an action proposal and decision as verified in service integration tests.
Sources: backend/tests/service.test.ts:114-138
UI Consistency Checks
QA for the frontend is governed by the DESIGN.md checklist and the shell-layout.test.ts suite.
Layout Invariants
Tests ensure that the Electron shell adheres to specific structural rules:
- One Home: Exactly one occurrence of settings, activity feed, and command form.
- No Duplicates: Prevents the rendering of legacy drawer navigation or redundant control bars.
- Compact Chrome: Ensures the custom titlebar height is exactly
38pxandtitleBarStyleis hidden.
Sources: tests/shell-layout.test.ts:15-38
Design Primitive Requirements
Automated or manual checks ensure that:
- No native HTML
title=attributes are used on buttons (clashes with themed<Tip>). - Z-index values follow the established ladder (e.g.,
--z-modal-backdropvs--z-setup). - Buttons use specific variants (e.g.,
default,ghost,secondary) rather than one-off styles.
Sources: instructions/DESIGN.md:85-115, instructions/DESIGN.md:148-160
QA Implementation Guidelines
| Principle | Requirement | Source File |
|---|---|---|
| Optimistic Rolling | Direct manipulation must paint first, then roll back on failure. | DESIGN.md:200-205 |
| Authority | State must be stored with its authority (Backend for shared facts). | AGENTS.md:33-40 |
| Seam Integrity | Tests must exercise the real path for resolvers and adapters. | AGENTS.md:147-150 |
| Privacy-First | Default mute state (muted: true) must be verified on startup. | packages/shared/tests/contracts.test.ts:135-140 |
The testing strategy ensures that J.A.R.V.I.S. remains a high-performance system by avoiding layout thrash and expensive Content-Mounts mid-gesture, verified through both logic tests and architectural constraints.
Sources: instructions/DESIGN.md:210-215, instructions/AGENTS.md:135-140