Screen & Camera Observation
Relevant source files
The following files were used as context for generating this wiki page:
The Screen & Camera Observation system provides the visual "eyes" for the AI tutor, enabling real-time context awareness of a user's environment and digital workspace. This system orchestrates media capture from local hardware (webcams) and operating system interfaces (screen sharing) to feed visual data into large language models (LLMs) for proactive tutoring and reactive feedback.
At its core, the system utilizes a "thick client" architecture where the browser handles media stream management, frame sampling, and initial processing, while the backend agent manages vision-language (VL) processing and decision-making for proactive interventions. This enables features like "Study Tutor" coaching, where the AI observes a student's progress unprompted and intervenes when struggle is detected.
Sources: apps/web/src/lib/live/useLiveSession.ts:24-30, services/agent/src/live/session.ts:86-90
Visual Data Capture Architecture
The system uses the CameraCapture class to manage MediaStream objects from both getUserMedia (camera) and getDisplayMedia (screen). It implements a rolling buffer strategy to ensure that the AI always has access to recent visual context without requiring constant high-bandwidth streaming.
Capture Strategies
The system differentiates between standard sampling and high-resolution "looks":
- Standard Sampling: Captures frames at approximately 1 frame per second (fps). For cameras, the long edge is capped at 640px. For screens, this is increased to 900px to ensure text legibility while maintaining performance.
- Hi-Res "Look": Triggered on-demand (e.g., via the
looktool), capturing frames at up to 1600px with 92% JPEG quality to read small labels or complex equations.
Sources: apps/web/src/lib/live/cameraCapture.ts:8-16, apps/web/src/lib/live/cameraCapture.ts:57-65
Media Flow Diagram
The following diagram illustrates the flow of visual data from the hardware to the backend agent.
This diagram shows how the CameraCapture maintains a buffer that useLiveSession draws from during turns, and how the server can explicitly request higher resolution frames via the look tool.
Sources: apps/web/src/lib/live/cameraCapture.ts:8-50, services/agent/src/live/session.ts:160-180, packages/shared/src/live-events.ts:50-80
Proactive Observation (Study Tutor)
The "Study Tutor" feature uses a proactive observation loop to monitor the student's screen without user prompts. This loop is managed by the tickObserve function in the client and the runObserve method in the server.
Observation Logic
The observation loop follows a specific lifecycle to avoid interrupting the user or performing redundant work:
- Interval Ticking: The client triggers an observation check at intervals based on the user's
interruptLevel(Quiet, Balanced, or Active). - State Gating: Observations are skipped if the user is currently speaking, the agent is thinking/speaking, or a "Focus" prompt is already visible.
- Visual Summary: The agent generates a dense factual summary of the screen (problem type, text, student actions).
- Intervention Decision: The agent decides whether to stay silent or speak a Socratic hint based on the student's progress and the
INTERRUPT_COOLDOWNconfiguration.
Sources: apps/web/src/lib/live/useLiveSession.ts:110-125, services/agent/src/live/session.ts:25-35, services/agent/src/live/session.ts:240-260
Configuration Table
| Setting | Level | Cooldown | Max per Min | Description |
|---|---|---|---|---|
Quiet | Low | 45,000ms | 2 | Only intervenes when the student is clearly stuck. |
Balanced | Med | 20,000ms | 4 | Hints when progress stalls. |
Active | High | 10,000ms | 6 | Checks in frequently to guide the student. |
Sources: services/agent/src/live/session.ts:16-23, apps/web/src/components/live/LiveStage.tsx:102-110
Vision-Language (VL) Processing
The system handles visual processing through two primary pathways: inline frame attachment and dedicated vision descriptions.
The look Tool Handshake
When the LLM requires high-resolution detail that standard sampling cannot provide, it invokes the look tool. This initiates a specific handshake:
- Agent sends
t: "need_frame"with areqId. - Client receives the request, captures a fresh 1600px frame, and replies with
t: "frame_response". - The agent resolves the pending
looktool execution with the new image data.
Sources: services/agent/src/live/session.ts:107-112, services/agent/src/live/session.ts:403-415, packages/shared/src/live-events.ts:31-33
Screen Context Memory
To maintain multi-turn context, the server stores a screenMemory buffer of the last 3 dense screen summaries. When a user asks a question, these summaries are injected into the prompt as "Ground Truth" to provide the LLM with context about what it has previously seen.
Implementation Details
Frame Sampling Implementation
The following snippet demonstrates how the system captures and scales images while respecting the hardware's native resolution to avoid upscaling artifacts.
// apps/web/src/lib/live/cameraCapture.ts:92-106
private async grab(size: number, q: number): Promise<ArrayBuffer | null> {
const v = this.video;
if (!v.videoWidth || !v.videoHeight) return null;
// Cap the long edge but never upscale past native (blurry garbage for VL).
const long = Math.max(v.videoWidth, v.videoHeight);
const scale = Math.min(1, size / long);
const w = Math.max(1, Math.round(v.videoWidth * scale));
const h = Math.max(1, Math.round(v.videoHeight * scale));
this.canvas.width = w;
this.canvas.height = h;
const ctx = this.canvas.getContext("2d");
if (!ctx) return null;
ctx.drawImage(v, 0, 0, w, h);
return new Promise((res) => this.canvas.toBlob((b) => (b ? b.arrayBuffer().then(res) : res(null)), "image/jpeg", q));
}
Sources: apps/web/src/lib/live/cameraCapture.ts:92-106
Detection of Visual Cues
The system uses regex-based heuristic checks to determine if a user's utterance requires a fresh "look" at the screen, forcing a bypass of the cached summaries.
Sources: services/agent/src/live/session.ts:50-55
Conclusion
The Screen & Camera Observation system effectively bridges the gap between the user's physical/digital environment and the AI's reasoning capabilities. By balancing proactive observation with reactive high-resolution "looks," and maintaining a rolling memory of visual summaries, the project creates a context-aware tutoring experience that minimizes latency and bandwidth while maximizing educational utility.