A2UI & Canvas System
Relevant source files
The following files were used as context for generating this wiki page:
The A2UI (Agent-to-User Interface) and Canvas System within Clawdbot provides a declarative framework for AI agents to generate and present rich, interactive user interfaces across different node platforms (macOS, iOS, Android). A2UI moves away from sending executable code, instead utilizing a declarative JSON format that describes the intent of a UI, which the client application then renders using native components.
The system is designed to be "safe like data, but expressive like code," ensuring security by preventing the execution of arbitrary LLM-generated scripts while allowing agents to build bespoke forms, dashboards, and interactive demos. The Canvas acts as the hosting environment for these interfaces, bridging the gap between the backend Agent and the frontend Node apps.
Sources: vendor/a2ui/README.md:3-33, skills/canvas/SKILL.md:3-10
System Architecture
The architecture consists of three primary layers: the Canvas Host, the Node Bridge, and the Node Apps. The Canvas Host serves the content via HTTP/WebSocket, while the Node Bridge (part of the Gateway) communicates URLs and state to connected nodes.
The Canvas Host Server specifically manages static files and live-reloading capabilities, while the Node Bridge ensures the correct URLs (often derived from Tailscale or LAN IPs) are reachable by the target hardware.
Sources: skills/canvas/SKILL.md:14-25, src/canvas-host/server.ts:250-280
A2UI: Agent-to-User Interface
A2UI is a streaming protocol and declarative data format optimized for updateable agent-generated UIs. It treats UI as a flat list of components with ID references, making it easy for Large Language Models (LLMs) to generate and update incrementally.
Core Philosophies
| Philosophy | Description |
|---|---|
| Security First | Declarative JSON format only; client applications maintain a catalog of trusted components. |
| LLM-friendly | Optimized for incremental updates and progressive rendering. |
| Framework Agnostic | Separates structure from implementation (Web, Flutter, SwiftUI, etc.). |
| Flexibility | Uses a "Smart Wrapper" pattern to connect existing UI components to the A2UI data binding system. |
Sources: vendor/a2ui/README.md:36-65
Data Binding and Actions
A2UI utilizes a SignalA2uiMessageProcessor to handle incoming messages and maintain the state of various "surfaces." When a user interacts with a component, the ClawdbotA2UIHost captures the event, resolves the relevant data path (data binding), and sends a userAction payload back to the native bridge.
Canvas Host Server
The Canvas Host is an HTTP and WebSocket server that serves static assets from a designated root directory (default: ~/clawd/canvas). It includes a specialized handleA2uiHttpRequest to serve the A2UI renderer assets and an injectCanvasLiveReload function to facilitate real-time development.
Key Components
- Static File Serving: Serves HTML, CSS, and JS from the
canvasHost.root. - Live Reload: Uses
chokidarto watch the filesystem and notifies connected canvases via WebSocket (/__clawdbot/ws) to reload on changes. - A2UI Asset Routing: Specifically routes requests starting with
/__clawdbot__/a2uito the internal A2UI renderer bundle.
Sources: src/canvas-host/server.ts:35-70, src/canvas-host/a2ui.ts:7-25
Native Integration (macOS)
On macOS, the CanvasManager handles the lifecycle of the canvas window. It manages the presentation of the CanvasWindowController, handles auto-navigation to the A2UI host, and provides methods for JavaScript evaluation and snapshotting.
Canvas Lifecycle Management
The CanvasManager follows a singleton pattern and observes the Gateway connection for updates to the canvasHostUrl.
Host URL Resolution
The system determines the correct URL based on the gateway.bind setting. It prioritizes Tailscale hostnames to ensure cross-device connectivity.
- Internal Prefix:
/__clawdbot__/canvas/ - A2UI Specific Path:
/__clawdbot__/a2ui/
Sources: skills/canvas/SKILL.md:27-35, apps/macos/Sources/Clawdbot/CanvasManager.swift:175-180
Interaction Bridge
To facilitate communication between the webview content and the native host, Clawdbot injects a bridge helper. This helper provides window.clawdbotSendUserAction, which is used by both custom HTML and the A2UI bootstrap.
// Example of the injected bridge helper
function sendUserAction(userAction) {
const id = userAction.id || globalThis.crypto.randomUUID();
const action = { ...userAction, id };
return postToNode({ userAction: action });
}
Sources: src/canvas-host/a2ui.ts:125-135, src/canvas-host/server.ts:120-140
Summary
The A2UI & Canvas System provides a robust, secure, and cross-platform method for agents to interact with users through rich interfaces. By combining a declarative UI protocol (A2UI) with a versatile web-hosting environment (Canvas), the system enables complex workflows like dynamic data collection and interactive dashboards while maintaining strict security boundaries and framework portability.