Action Engine & Automation

Relevant source files

The following files were used as context for generating this wiki page:

• backend/src/action-engine.ts
• backend/src/cursor-bridge.ts
• backend/src/browser-adapter.ts
• backend/src/handler.ts
• src/renderer.tsx
• backend/tests/action-engine.test.ts
• instructions/DESIGN.md

The Action Engine serves as the central orchestration layer for executing system tasks, browser automation, and application control within the J.A.R.V.I.S. ecosystem. It is designed with a strict "human-in-the-loop" philosophy, ensuring that no sensitive or irreversible system changes occur without explicit user authorization.

The engine manages the lifecycle of a JarvisActionIntent, moving from a proposed state to execution only after user validation. It bridges high-level AI reasoning with low-level OS capabilities such as file system operations, media control, and cursor manipulation.

Sources: backend/src/action-engine.ts, instructions/DESIGN.md:37-41, backend/src/handler.ts:327-340

Core Architecture

The automation system is split between the backend service, which manages state and capability routing, and the desktop renderer, which handles the user interface for approvals.

The Action Lifecycle

Every automation request follows a structured state machine to maintain safety and auditability.

The diagram above illustrates the transition states of an action intent, emphasizing the mandatory user decision gate.
Sources: backend/src/action-engine.ts:5-12, src/renderer.tsx:645-680

Key Components

Component	Responsibility
JarvisActionEngine	Interface defining the contract for proposing and deciding actions.
DefaultJarvisActionEngine	Implementation managing the in-memory store of intents and executing adapters.
Cursor Bridge	Low-level integration for system-wide mouse movement, clicking, and scrolling.
Browser Adapter	Handles web search and content fetching for research automation.
Tool Loop	Backend logic that executes a chain of tool calls until a final content result is reached.

Sources: backend/src/action-engine.ts, backend/src/cursor-bridge.ts, backend/src/handler.ts:474-560

Automation Capabilities

The system exposes a variety of "Capabilities" that can be invoked via tool-calling or direct commands. These are categorized by their impact on the system.

System & App Control

Sensitive operations like shutting down the PC or opening folders are restricted by allowlists. For example, system.open_folder will fail if the requested path is not in a safe list (e.g., C:\Windows\System32 is blocked).

// Example of safety logic in DefaultJarvisActionEngine
if (intent.capability === "system.open_folder") {
  const folder = String(intent.params?.folder ?? "");
  const allowlist = ["desktop", "downloads", "documents", "pictures", "videos", "musik"];
  if (!allowlist.includes(folder.toLowerCase())) {
    throw new Error(`Pfad '${folder}' ist nicht in der sicheren Allowlist.`);
  }
}

Sources: backend/src/action-engine.ts:98-105, backend/tests/action-engine.test.ts:46-58

Media & Interface Control

Automation also extends to environmental control, such as managing media playback or interacting with the "Barehands" hand-tracking interface.

• media.control: Supports play, pause, stop, next, prev, volup, voldown.
• barehands.board: Places cards, images, or 3D models onto the hand-tracking glass board.
• barehands.cursor: Enables system-wide mouse control via hand gestures.

Sources: backend/src/handler.ts:219-270, src/renderer.tsx:1120-1145

Execution Flow

When a tool is triggered by the AI (e.g., Grok), the backend generates an intent. This intent is fanned out as a JarvisLiveEvent over a Server-Sent Events (SSE) stream to the Electron renderer.

This sequence demonstrates how a voice or chat command translates into a visual UI update on the "Main Stage" after backend processing.
Sources: backend/src/handler.ts:580-605, src/renderer.tsx:1250-1280, backend/src/action-engine.ts:55-80

Safety Constraints

The Action Engine implements several layers of validation to prevent misclassification and malicious execution:

1. Misclassification Blocking: Prevents websites (like Wikipedia) from being launched as Windows apps via app.open_app.
2. Explicit Approval: High-risk actions like system.shutdown or system.cursor_click stay in the proposed state indefinitely until the user interacts with the UI.
3. Bounded Arguments: All tool parameters are typed and validated at the boundary before being passed to the execution logic.

Sources: backend/tests/action-engine.test.ts:18-44, instructions/design-backend-prompt.txt:130-140

Conclusion

The Action Engine and Automation system in J.A.R.V.I.S. provide a robust framework for desktop control that prioritizes user safety and intent. By decoupling the proposal of an action from its execution and requiring a human-in-the-loop decision, the system allows for powerful automation while mitigating the risks associated with autonomous AI execution on a local machine.