Wire Protocol & SSE
Relevant source files
The following files were used as context for generating this wiki page:
The OpenLive project utilizes a specialized wire protocol for real-time communication between the client (browser/desktop app) and the agent service. This protocol is primarily built upon a Thick Client architecture where the browser handles resource-intensive tasks like Voice Activity Detection (VAD), Speech-to-Text (STT), and Text-to-Speech (TTS) on-device. The wire protocol itself carries text, control signals, and visual data (camera/screen frames) via WebSockets and Server-Sent Events (SSE).
Sources: packages/shared/src/live-events.ts:4-10, apps/web/src/lib/live/useLiveSession.ts:25-30
Communication Architecture
The system employs a dual-channel approach for communication. The main live interaction occurs over a WebSocket (WS) connection, while the internal logic and provider adapters (like Anthropic or OpenAI) utilize Server-Sent Events (SSE) which are then wrapped and tunneled through the WebSocket to the client.
Protocol Frame Types
The WebSocket connection supports two primary data formats:
- Binary Frames: Prefixed with a 1-byte tag (e.g.,
0x02forFRAME_IN) to transmit JPEG camera or screen frames efficiently. - Text Frames: JSON-encoded discriminated unions representing structured messages like user text, session configurations, or wrapped SSE events.
Sources: packages/shared/src/live-events.ts:13-16, services/agent/src/live/session.ts:145-155
Data Flow Overview
This diagram illustrates the flow of messages between the client's voice engine and the server's LLM agent.
Server-Sent Events (SSE) Schema
The SseEvent is a discriminated union used to stream granular updates from the agent to the client. These events are often generated by external AI providers and normalized by the system's harness.
Standard SSE Event Types
| Event Type | Purpose | Data Payload |
|---|---|---|
text_delta | Streaming response text | text: string |
reasoning_delta | Streaming AI "thinking" process | text: string |
tool_start | Indicates a tool is being invoked | id: string, tool: string, summary?: string |
tool_done | Indicates tool execution finished | id: string, detail?: string |
todos | Updates a checklist in the UI | items: { text: string, done: boolean }[] |
usage | Token count and cost reporting | contextTokens, outputTokens, costUsd |
done | Final signal for a completion turn | N/A |
Sources: packages/shared/src/sse-events.ts:11-30
WebSocket Wire Protocol
The LiveClientMsg and LiveServerMsg schemas define the JSON structure of text-based WebSocket frames.
Client to Server Messages
- user_text: Sent when on-device STT completes a turn. Includes atomic base64 frames for camera or screen.
- session_config: Sets
studyGoalandinterruptLevel. - observe: Triggered by the proactive study tutor to provide visual context without user speech.
- cancel: Barge-in signal sent when the user starts talking over the agent. Includes
spokentext for accurate history truncation. - frame_response: Replies to a server request for a high-resolution frame.
Sources: packages/shared/src/live-events.ts:39-71
Server to Client Messages
- sse: Wraps a standard
SseEvent. This allows the client to reuse the same reducer logic for both chat and live modes. - need_frame: Handshake requesting a fresh hi-res frame from the client for tools like
look. - tool_bridge: Requests the client to perform OS-level actions (e.g.,
clipboard_read,open_url) via the Electron bridge.
Sources: packages/shared/src/live-events.ts:18-28, services/agent/src/live/session.ts:200-215
Provider Harness Mapping
The packages/harness module contains adapters that translate raw SSE streams from various providers into the canonical ProviderEvent format.
Interaction: Barge-in and Truncation
When a cancel message is received via the wire protocol, the server must handle history accurately. The spoken field contains the text the client actually voiced before the user interrupted. The server uses this to truncate the saved assistant response.
function truncateSpokenText(blocks: MessageBlock[], spoken: string): void {
const s = spoken.trim();
let placed = false;
for (const b of blocks) {
if (b.type !== "text") continue;
if (!placed) { b.text = s; placed = true; } else b.text = "";
}
if (!placed && s) blocks.unshift({ type: "text", text: s });
}
Sources: services/agent/src/live/session.ts:32-42, packages/shared/src/live-events.ts:61-64
Visual Data Handling
The protocol treats visual data as a primary citizen. Frames can be sent:
- Reactively: Attached to
user_textturns as base64 strings in the JSON payload. - Proactively: Via the
observemessage. - On-Demand: Through the
need_frame(server) andframe_response(client) handshake, often triggered by thelooktool.
Sources: packages/shared/src/live-events.ts:44-48, services/agent/src/live/session.ts:500-515, apps/web/src/lib/live/useLiveSession.ts:275-290
Summary
The Wire Protocol and SSE system form the backbone of OpenLive's low-latency interaction model. By combining binary WebSocket frames for images, structured JSON for control, and tunneled SSE for streaming LLM responses, the architecture supports complex behaviors like barge-in, proactive observation, and tool-assisted OS interaction while maintaining a consistent data format across different AI providers.