Live VNC Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
Live VNC Monitoring is a critical diagnostic and observation system within the project that allows developers and operators to view and interact with automation agents in real-time. By leveraging Virtual Network Computing (VNC) over WebSockets, the system provides a visual window into the automated browser instances running on the server, facilitating immediate feedback on workflow execution and troubleshooting.
The system is integrated directly into the frontend React application, providing a grid view of all active sessions and a detailed "Focus Mode" for individual sessions. This focused view synchronizes live browser streams with real-time log execution, enabling a comprehensive understanding of the agent's state and actions. Sources: frontend/src/features/vnc/containers/VncPageContainer.tsx, frontend/src/features/vnc/containers/VncSessionPageContainer.tsx, AGENTS.md:29
System Architecture and Data Flow
The VNC monitoring system operates through a distributed architecture involving the React frontend, an Express API server, and the automation runners. The server manages display allocation and exposes VNC ports, which the frontend consumes via specific WebSocket URLs.
Connection Workflow
The following diagram illustrates the lifecycle of a VNC monitoring session from discovery to interactive control.
The system uses display_allocated and display_released messages over a central WebSocket to track active browser instances. Sources: frontend/src/hooks/useWebSocket.ts:25-26, frontend/src/features/vnc/containers/VncPageContainer.tsx:18
Key Components
| Component | Description | Source |
|---|---|---|
VncPageContainer | Orchestrates the grid view of all active live sessions. | frontend/src/features/vnc/containers/VncPageContainer.tsx:13 |
VncSessionPageContainer | Manages the detailed view of a single session, including logs and VNC stream. | frontend/src/features/vnc/containers/VncSessionPageContainer.tsx:28 |
VncViewer | A lazy-loaded component (via @novnc/novnc) that renders the remote desktop stream. | frontend/src/features/vnc/containers/VncSessionPageContainer.tsx:21 |
LogsViewer | Displays real-time execution logs filtered by workflowId and profileName. | frontend/src/components/shared/LogsViewer.tsx:14 |
useWebSocket | A hook managing the real-time message stream (logs, status, display events). | frontend/src/hooks/useWebSocket.ts:167 |
Session Management
Sessions are identified by a combination of workflowId and profileName. The server communicates availability through the DisplaySession data structure, which includes the assigned VNC port and display number.
Display Allocation Data Structure
The WebSocket message protocol includes specific fields for display management:
vnc_port/vncPort: The network port on the server where the VNC server is listening.display_num/displayNum: The X11 display number assigned to the browser instance.profileName: The specific Instagram profile associated with the session.
Sources: frontend/src/hooks/useWebSocket.ts:38-42, frontend/src/features/vnc/containers/VncSessionPageContainer.tsx:142
Interactive Handoff Logic
To prevent accidental interference with automation agents, the VNC stream defaults to a "locked" or read-only state. Users must explicitly request control.
Real-time Log Integration
A defining feature of the monitoring system is the synchronization between the VNC visual stream and the LogsViewer. The LogsViewer uses the useWebSocket hook to filter for messages specific to the active session's workflowId and profileName.
Log Message Parsing
Messages received via WebSocket with the type log are parsed into a LogEntry structure:
function parseLogEntry(data: WebSocketMessage, currentProfile: string | null): LogEntry {
return {
message: data.message!,
level: data.level || 'info',
source: data.source || 'unknown',
workflowId: (data.workflowId ?? data.workflow_id) ?? undefined,
profileName: data.profileName || currentProfile || undefined,
ts: Date.now(),
};
}
Sources: frontend/src/hooks/useWebSocket.ts:60-74, frontend/src/components/shared/LogsViewer.tsx:20
Mobile vs Desktop Layouts
The monitoring interface is responsive, adapting the layout based on the device's screen size to ensure usability in field conditions.
Layout Differences
| Feature | Desktop Layout | Mobile Layout |
|---|---|---|
| Panels | Horizontal split (Resizable) | Vertical stack (Scrollable) |
| Log Visibility | Always visible in side panel | Toggled via "Show/Hide Logs" button |
| Header | Full breadcrumbs and status | Compact with profile name focus |
| Control | Overlay-based confirmation | In-line button transitions |
Sources: frontend/src/features/vnc/containers/VncSessionPageContainer.tsx:158-185, frontend/src/features/vnc/containers/VncSessionPageContainer.tsx:219-245
Server-Side Implementation
The server-side implementation in server/index.ts initializes the WebSocket server and handles the routing for display management. It also includes cleanup logic to ensure that orphaned VNC processes or displays are released if the server restarts unexpectedly.
- WebSocket Initialization:
initWebSocket(server)handles the upgrade of HTTP connections to WebSocket connections. - Cleanup:
cleanupOrphanedProcesses()andprofileManager.reconcileRuntimeStatuses()ensure that sessions are accurately reflected after a crash. - Auth: Connections to the VNC WebSocket are protected via Clerk authentication tokens passed as a query parameter (
?token=...).
Sources: server/index.ts:32, server/index.ts:133-143, frontend/src/hooks/useWebSocket.ts:121-125
The Live VNC Monitoring system provides an essential bridge between automated backend processes and human oversight, ensuring that complex automation workflows can be monitored and managed with high precision.