VNC Integration

Relevant source files

The following files were used as context for generating this wiki page:

• docker-compose.yml
• supervisord.conf
• README.md
• webui.py
• src/utils/utils.py

The VNC Integration in the web-ui project provides a real-time visual interface for monitoring and interacting with the automated browser agent. This system is primarily designed for Docker-based deployments, allowing users to observe the AI's browser interactions through a web-based noVNC viewer or a direct VNC client.

The integration serves as a bridge between the headless browser execution environment and the end-user, facilitating debugging and transparency of the agent's actions. It encapsulates a full display stack including a virtual frame buffer (Xvfb), a VNC server (x11vnc), and a WebSocket proxy (noVNC) to stream the display to standard web browsers.

Sources: README.md:124-129, docker-compose.yml:5-13

System Architecture

The VNC stack operates as a multi-layered service managed by supervisord within a Linux container. It ensures that a graphical environment is available for Chromium even when no physical monitor is attached.

Display Stack Components

1. Xvfb (X Virtual Frame Buffer): Creates a virtual display in memory (default :99) where the browser renders its GUI.
2. x11vnc: A VNC server that captures the content of the Xvfb virtual display and makes it available over the network via the RFB protocol.
3. noVNC: A VNC client implementation using HTML5 Canvas and WebSockets, allowing the VNC stream to be viewed in a web browser without additional software.

The following diagram illustrates the data flow from the Browser Agent to the User Interface through the VNC components:

A brief sentence explaining what the diagram shows: This diagram shows the internal container process flow where browser rendering is captured by a VNC server and proxied to the user through web-friendly protocols.

Sources: supervisord.conf:10-75, docker-compose.yml:10-13

Configuration and Connectivity

The VNC system is highly configurable via environment variables, allowing for custom resolutions and security credentials.

Port Mappings

Service	Container Port	Host Port	Protocol	Description
noVNC Interface	6080	6080	TCP/HTTP	Web interface for viewing the browser
Direct VNC	5901	5901	TCP/RFB	Direct connection for VNC clients
Remote Debug	9222	9222	TCP/HTTP	Chrome remote debugging port

Sources: docker-compose.yml:10-13, README.md:125-128

Environment Variables

Configuration is handled through the .env file or direct container environment settings:

• VNC_PASSWORD: Sets the authentication password for the VNC session. Defaults to vncpassword.
• RESOLUTION: Defines the virtual screen dimensions and color depth (e.g., 1920x1080x24).
• DISPLAY: The X11 display identifier, typically set to :99.

Sources: docker-compose.yml:34-40, supervisord.conf:11-15

Service Management

The supervisord.conf file defines the lifecycle and dependencies of the VNC-related processes to ensure the stack initializes in the correct order.

Initialization Sequence

This sequence diagram outlines the strict dependency chain managed by Supervisor to ensure the display environment is ready before the browser launches.

Sources: supervisord.conf:10-95

Process Details

• vnc_setup: A one-time task that creates the .vnc directory and writes the encrypted password file using the vncpasswd utility.
• x11vnc: Configured with the -forever flag to stay active after client disconnects and -rfbauth to enforce the configured password.
• novnc: Runs the novnc_proxy which listens on 0.0.0.0:6080 and redirects traffic to the local VNC port 5901.

Sources: supervisord.conf:23-64

Real-time Monitoring via WebUI

While VNC provides a full interactive desktop view, the application also includes a "Live Browser View" within the Gradio interface.

In webui.py, if the agent is running in headless mode, the system periodically captures screenshots from the browser context and updates an HTML component to provide a near real-time stream without requiring a separate VNC client.

# From webui.py:534-541
while not agent_task.done():
    try:
        encoded_screenshot = await capture_screenshot(_global_browser_context)
        if encoded_screenshot is not None:
            html_content = f'<img src="data:image/jpeg;base64,{encoded_screenshot}" style="width:{stream_vw}vw; height:{stream_vh}vh ; border:1px solid #ccc;">'

Sources: webui.py:534-541, src/utils/utils.py:192-228

Conclusion

VNC Integration is a critical component for the Dockerized version of the web-ui, providing the necessary infrastructure for visual feedback in headless environments. By combining Xvfb, x11vnc, and noVNC, the project allows developers to monitor AI behavior through standard web browsers at http://localhost:6080/vnc.html, ensuring that automated browser tasks remain observable and debuggable.