Agent WebSocket Server
Relevant source files
The following files were used as context for generating this wiki page:
The Agent WebSocket Server serves as the real-time communication backbone of the OpenLive system, facilitating low-latency interaction between the client-side voice engine and the server-side AI agent. It acts as a "thin" proxy, meaning the client handles heavy tasks like Voice Activity Detection (VAD) and Speech-to-Text (STT), while the server focuses on LLM orchestration, tool execution, and session persistence.
The server operates on a protocol that exchanges JSON messages for control and state, alongside binary frames for visual data such as camera JPEGs. It manages high-level features like proactive "observations" (where the agent looks at the user's screen unprompted) and "delegation" to sub-agents for complex web-based tasks.
Sources: packages/shared/src/live-events.ts:1-12, services/agent/src/server.ts:14-25
Server Architecture and Lifecycle
The server is built using Hono and the Node.js http module. It listens on a configurable port (defaulting to 8787) and implements a shared-secret authorization gate to ensure only authorized clients can establish a connection.
Core Components
- Hono Server: Handles initial HTTP requests, CORS, and health checks.
- WebSocket Upgrade: The
attachLiveWsfunction upgrades incoming requests to the/livepath into persistent WebSocket connections. - LiveSession: Each connection instantiates a
LiveSessionobject which tracks the conversation history, active frames, and tool handshakes.
Sources: services/agent/src/server.ts:14-30, services/agent/src/live/session.ts:58-80
Connection Flow
The following sequence diagram illustrates the handshake and initialization of a live session:
Wire Protocol (Live Events)
Communication is split between binary frames for media and JSON-encoded text frames for control logic.
Message Types
| Message Tag/Type | Direction | Description |
|---|---|---|
0x02 (LIVE_TAG.FRAME_IN) | Client -> Server | Binary JPEG camera or screen frame. |
user_text | Client -> Server | Final STT transcript + optional inline base64 frames. |
sse | Server -> Client | Wrapped Server-Sent Events (text deltas, tool starts, usage). |
need_frame | Server -> Client | Request for a high-resolution frame (for look tool). |
observe | Client -> Server | Proactive screen look frame without user speech. |
cancel | Client -> Server | Barge-in signal to abort in-flight LLM generation. |
Sources: packages/shared/src/live-events.ts:14-80, apps/web/src/lib/live/liveClient.ts:74-115
Session Management and Proactive Intervention
The LiveSession class manages the state of the interaction, including "Proactive Observations." The Agent is designed to "watch" the user's screen and intervene if it detects struggle or a need for guidance.
Proactive Observation Logic
The server implements an intervention policy based on the user's configured interruptLevel (quiet, balanced, or active).
Tool Execution and Delegation
The Agent utilizes tools defined in tools.ts. A unique feature is the delegate tool, which hands off complex research tasks to a "worker" sub-agent. This worker runs its own tool loop (e.g., web_search, fetch_url) while the main agent provides verbal updates (narrations) to the user.
- Main Tools:
delegate,update_todos,remember. - Worker Tools:
web_search,fetch_url. - Handshake Tools:
look(requests frame from client),clipboard_read/write,open_url.
Sources: services/agent/src/tools.ts:100-145, services/agent/src/live/worker.ts:40-75
Implementation Details: LiveSession Handshakes
The server often needs to request data back from the client mid-turn (e.g., when the LLM uses the look tool). This creates an asynchronous handshake:
// services/agent/src/live/session.ts:355-368
private requestFrame(): Promise<Frame | null> {
return new Promise((resolve) => {
const reqId = randomUUID();
const timer = setTimeout(() => {
if (this.lookPending?.reqId === reqId) {
this.lookPending = null;
resolve(null);
}
}, 8_000);
this.lookPending = { reqId, resolve: (f) => { clearTimeout(timer); resolve(f); } };
this.send({ t: "need_frame", reqId });
});
}
Barge-in Handling
When a user starts speaking over the agent, the client sends a cancel message. The server immediately aborts the AbortController associated with the current LLM turn and truncates the saved conversation text to match only what was actually voiced by the client before the interruption.
Sources: services/agent/src/live/session.ts:27-30, services/agent/src/live/session.ts:290-302
Conclusion
The Agent WebSocket Server is a specialized real-time relay that enables the "thick client" voice architecture. By offloading media processing to the client and providing a structured JSON/Binary protocol, the server maintains low latency and high reliability for AI-driven learning assistance and tool-based interactions.