AI Agents & Assistant Mode
Relevant source files
The following files were used as context for generating this wiki page:
AI Agents and Assistant Mode represent an experimental extension of the mausVoice platform, shifting the application from a passive dictation utility to an active, autonomous assistant. While standard dictation focuses on transcribing and cleaning up text, Assistant Mode follows dictated commands to write, edit, or transform text and interact with the local desktop environment.
Sources: apps/desktop/src/i18n/locales/en.json, apps/desktop/PRODUCT.md:12-14, README.md:37-41
Architectural Overview
The agent system is built on a "TypeScript as the Brain" architecture, where all decision-making logic resides in the TypeScript layer while native capabilities are exposed through a bridge. The Agent Mode is designed to be autonomous, gathering context from the user's screen or focused text field before executing actions through specialized tools.
Sources: apps/desktop/README.md:38-42, packages/agent/scripts/example.ts:133-145, llms.txt:30-38
The Agent Loop
The core of the system is the AgentLoop, which manages iterations of thinking and tool execution. It follows a specific workflow to accomplish tasks:
- Gather Context: Uses tools to read what the user is looking at (e.g., "this email").
- Execute Work: Performs the requested task (e.g., running a terminal command).
- Deliver Results: Pastes the final output directly into the user's focused application.
Sources: packages/agent/scripts/example.ts:133-145, packages/agent/scripts/example.ts:162-170
The following diagram illustrates the lifecycle of a single user request within the Agent Loop:
Key Features & Capabilities
Assistant Mode offers several advanced features that differentiate it from standard AI-cleaned dictation.
| Feature | Description |
|---|---|
| Autonomous Action | Takes full action (writing, running commands) instead of just displaying output. |
| Context Awareness | Reads accessibility context and takes screenshots to understand what the user refers to. |
| Power Mode | Experimental feature allowing the assistant to run terminal commands on the user's behalf. |
| Direct Injection | Pastes results directly into the active field using paste_text instead of a clipboard buffer. |
| Multi-Iteration | Refines output across multiple steps to ensure the task is fully completed. |
Sources: apps/desktop/src/i18n/locales/en.json, packages/agent/scripts/example.ts:133-145, AGENTS.md:1-10
Tool System & Native Integrations
The agent interacts with the system through a set of defined AgentTool interfaces. Each tool includes a description that the LLM uses to decide when to invoke it.
Sources: packages/agent/scripts/example.ts:50-131
Tool Definitions
read_accessibility_context: Reads the content of the focused text field, cursor position, and the name of the focused application.paste_text: Natively injects text into the focused field, fulfilling the "Deliver Result" phase of the workflow.grab_screenshot: Captures the current screen to provide visual context for UI elements beyond the text field.execute_terminal_command: Runs shell commands. This requires explicit user confirmation for each command as a security guardrail.
Sources: packages/agent/scripts/example.ts:50-120, apps/desktop/src/i18n/locales/en.json
Configuration and Security
Assistant Mode includes specific user-facing settings to manage its experimental nature and potential risks.
Security Guardrails
Because the agent can perform destructive actions via terminal commands, a "Power Mode" setting exists. This requires users to accept full responsibility for actions taken by the assistant. Additionally, a "confirmation phrase" or explicit approval dialog is required before terminal commands are executed.
Sources: apps/desktop/src/i18n/locales/en.json, packages/agent/scripts/example.ts:107-112
Configuration Options
| Option | Description |
|---|---|
| Agent Mode Toggle | Enables/Disables the experimental assistant features. |
| Assistant Hotkey | Customizable shortcut to activate the agent independently of standard dictation. |
| AI Provider | Selects which LLM (e.g., Groq, OpenAI) drives the agent's logic. |
| Power Mode | Allows terminal command execution (requires manual restart to apply). |
Sources: apps/desktop/src/i18n/locales/en.json, llms.txt:68-72
Usage Implementation Example
The implementation of the AgentLoop requires a provider for streaming chat and a set of tools. The loop emits events for each stage of the process, allowing the UI (such as the "pill" overlay) to update its state.
// Example from packages/agent/scripts/example.ts:162-203
const loop = new AgentLoop({
provider,
tools,
systemPrompt,
maxIterations: 20,
});
for await (const event of loop.run(conversationHistory)) {
switch (event.type) {
case "tool-call-start":
console.log(`Running ${event.toolName}...`);
break;
case "finish":
console.log("Task Completed");
break;
}
}
Sources: packages/agent/scripts/example.ts:162-203
Conclusion
AI Agents & Assistant Mode transition mausVoice from a dictation tool into a desktop companion capable of performing tasks autonomously. By leveraging accessibility APIs, screen capture, and terminal access, the system can understand the user's intent within their current context and deliver completed work directly into any application.
Sources: apps/desktop/PRODUCT.md, packages/agent/scripts/example.ts:133-145