Computer Use & UI Interaction Recording
Relevant source files
The following files were used as context for generating this wiki page:
The Computer Use and UI Interaction Recording system in Daytona provides a comprehensive suite of tools for AI agents and users to interact with and monitor a VNC-based desktop environment within a sandbox. This module enables automated control of mouse and keyboard inputs, real-time screen capturing, and the ability to record user or agent interactions for debugging, auditing, or training purposes.
The system is architected as a set of background processes managed by the Daytona Daemon, exposed through a specialized "Toolbox" API. By leveraging virtual framebuffers and standard Linux desktop components, it creates a headless yet fully functional graphical environment accessible via VNC and web-based NoVNC clients.
Sources: libs/computer-use/README.md, apps/daemon/pkg/toolbox/server.go:270-322
System Architecture
The Computer Use module manages a specific stack of processes to provide a stable graphical environment. These processes are started in a prioritized order to satisfy dependencies, ensuring that the display server is available before the desktop environment or VNC server attempts to launch.
Core Process Stack
| Process | Description | Priority |
|---|---|---|
| Xvfb | X Virtual Framebuffer: provides the virtual display (:1). | 100 |
| xfce4 | Desktop Environment: starts the graphical UI components. | 200 |
| x11vnc | VNC Server: exposes the virtual display via VNC protocol. | 300 |
| novnc | Web-based Client: provides browser access to the VNC server. | 400 |
Sources: libs/computer-use/README.md
Process Flow Diagram
The following diagram illustrates the dependency-based startup sequence and the management of these processes by the Computer Use manager.
Sources: libs/computer-use/README.md
Interaction API
The Toolbox API provides granular control over the virtual environment. AI agents use these endpoints to "see" the screen via screenshots and "act" via keyboard and mouse simulations.
Mouse and Keyboard Control
Interaction is powered by underlying system tools like xdotool and xautomation. The API supports complex gestures including clicking, dragging, scrolling, and multi-key hotkey combinations.
| Endpoint | Method | Description |
|---|---|---|
/computeruse/mouse/click | POST | Clicks a button at specified X/Y coordinates. |
/computeruse/mouse/drag | POST | Drags from start to end coordinates. |
/computeruse/keyboard/type | POST | Types text with optional delay between keys. |
/computeruse/keyboard/hotkey | POST | Executes combinations like ctrl+c. |
Sources: libs/toolbox-api-client-go/api/openapi.yaml:94-250, libs/computer-use/README.md
Visual Feedback and Screenshots
Agents can retrieve the current state of the UI through various screenshot methods. These include full-screen captures, specific region captures, and compressed formats (JPEG/PNG) to optimize bandwidth.
UI Interaction Recording
The recording system operates independently of the real-time interaction plugin, allowing for the capture of screen sessions into video files. This is managed by a dedicated RecordingService.
Recording Lifecycle
- Start: Initiates a capture session with an optional label.
- Management: Active recordings are tracked by unique IDs and can be listed or queried for status.
- Stop: Finalizes the video file and calculates metadata like duration and file size.
- Export: Completed recordings can be downloaded via the API.
| Endpoint | Description |
|---|---|
POST /computeruse/recordings/start | Starts a new screen recording session. |
POST /computeruse/recordings/stop | Stops an active session by ID. |
GET /computeruse/recordings | Lists all active and completed recordings. |
GET /computeruse/recordings/{id}/download | Downloads the recording as a binary stream. |
Sources: libs/toolbox-api-client-go/api/openapi.yaml:310-381, apps/daemon/pkg/toolbox/server.go:323-332
Implementation Details
Environment Configuration
The system requires specific environment variables to be set within the sandbox container to function correctly.
# Default Configuration for Computer Use
ENV DISPLAY=:1
ENV VNC_PORT=5901
ENV NO_VNC_PORT=6080
ENV VNC_RESOLUTION=1280x720
ENV VNC_USER=daytona
Sources: libs/computer-use/README.md
Toolbox Server Integration
The Daytona Daemon initializes a "Lazy Computer Use" implementation. This allows the Toolbox server to start immediately, while the heavy VNC plugin components are loaded in the background from a specific plugin path (usually /usr/local/lib/daytona-computer-use).
// apps/daemon/pkg/toolbox/server.go:270-285
lazyCU := computeruse.NewLazyComputerUse()
s.ComputerUse = lazyCU
go func() {
pluginPath := "/usr/local/lib/daytona-computer-use"
// ...
impl, err := manager.GetComputerUse(s.logger, pluginPath)
if err == nil {
lazyCU.Set(impl)
}
}()
Sources: apps/daemon/pkg/toolbox/server.go:270-285
Conclusion
The Computer Use and Recording system transforms a standard Daytona sandbox into a visually interactive environment. By providing high-level API abstractions over low-level X11 and VNC tools, it enables complex UI automation and comprehensive session auditing for both human users and AI agents.