Introduction to Nudge
Relevant source files
The following files were used as context for generating this wiki page:
Nudge is an AI-powered learning assistant designed to provide real-time, socratic guidance to students as they work. Unlike traditional AI assistants that may provide direct answers, Nudge focuses on identifying moments of struggle and offering concise hints to help students progress without taking over the work. It operates as a live voice tutor that "watches" the student's screen and intervenes only when necessary to maintain the student's learning momentum.
Sources: README.md:3-6, website/src/renderer.js:14-20
The system is built as a fork of OpenLive, utilizing local Ollama models for both vision (screen observation) and speech (spoken replies). It emphasizes restraint and context-awareness, staying quiet during productive study and offering well-timed prompts when hesitation or repeated errors are detected.
Sources: README.md:8-13, website/src/renderer.js:46-52
Core Architecture and Data Flow
Nudge employs a dual-model architecture to separate vision tasks from conversational tasks, ensuring low-latency interactions. The "Eyes" model processes screen content to generate context, while the "Talk" model handles fast spoken replies using that context.
Sources: README.md:15-20
Multi-Model Interaction Flow
The data flow involves capturing screen frames, summarizing them via the vision model, and passing those summaries to the talk model along with microphone input.
Spoken turns are optimized by avoiding re-sending full screen frames to the vision model for every interaction, relying instead on cached summaries.
Sources: README.md:46-52
System Layers
Nudge is structured into four primary layers that govern its behavior and user interaction:
| Layer | Responsibility |
|---|---|
| Signal Layer | Captures pauses, retries, and patterns of uncertainty to differentiate productive struggle from confusion. |
| Policy Layer | Determines the intervention strategy (wait, ask, hint, or scaffold) based on learner state and task complexity. |
| Interaction Layer | Manages the UI, ensuring short, legible responses that feel like a collaborator rather than a standard assistant. |
| Outcome Layer | Measures student progress, course correction, and understanding reached. |
| Sources: website/src/renderer.js:108-125 |
AI Persona and Tutoring Logic
The assistant follows a specific persona defined as a "warm, patient live tutor." The tutoring logic is strictly Socratic, prioritizing questions and hints over complete solutions.
Sources: services/agent/src/prompt.ts:5-17
Proactive Observation Rules
Nudge performs "Proactive Observe" turns where it peeks at the student's screen without being prompted. The behavior during these turns is governed by an "Interrupt Level":
- SILENCE: The default state when the student is making steady progress.
- Soft Nudge: A short sentence if the student is idle too long or repeating mistakes.
- Socratic Question: Preferred over explanations to unlock the next move.
Sources: services/agent/src/prompt.ts:27-41
Interrupt Levels
| Level | Behavior |
|---|---|
quiet | Speak up rarely; default to SILENCE unless clearly stuck. |
balanced | Speak when stuck or confused; stay quiet during progress. |
active | Check in more often with short hints when progress stalls. |
| Sources: services/agent/src/prompt.ts:50-59 |
User Interface and Experience
Nudge features a unique "Sphere UX" designed to minimize distraction during study sessions.
Sources: README.md:54-59
The Floating Sphere
When a session starts, the application UI collapses into a ~64px floating orb (MiniSphere) that stays always-on-top in the top-right corner. This allows students to focus on their primary work (e.g., PDFs, worksheets) while Nudge observes in the background.
Sources: README.md:54-59, apps/web/src/components/live/MiniSphere.tsx:32-37
Tooling and Capabilities
Nudge uses a delegated worker architecture to handle research and external data retrieval. The main agent delegates tasks to a "Worker" subagent, which manages web-based tools.
Sources: services/agent/src/tools.ts:162-177, services/agent/src/live/worker.ts:40-45
Core Tools
web_search: Searches the web for current or factual information.fetch_url: Retrieves readable text from specific public web pages.remember: Saves short facts (e.g., user preferences, name) into persistent memory (agent_notes) for future sessions.update_todos: Publishes or updates a multi-step checklist in the UI.look: Manually triggers a high-resolution screen capture to refresh the "Eyes" context.
Sources: services/agent/src/tools.ts:77-158, services/agent/src/prompt.ts:23-25
Web Search Flow
When a student asks a question requiring external data, the following sequence occurs:
Sources: services/agent/src/live/worker.ts:47-68
Technical Implementation Details
Live Session Management
The useLiveSession hook orchestrates the live call, including Voice Activity Detection (VAD), Speech-to-Text (STT), and Text-to-Speech (TTS). It manages the "Observe Loop" which captures screen frames at intervals defined by the interrupt level.
Sources: apps/web/src/lib/live/useLiveSession.ts:39-65, apps/web/src/lib/live/useLiveSession.ts:133-149
Screen Capture
High-resolution captures (1600px width, 0.92 JPEG quality) are sent to the vision model when specific details are needed via the onNeedFrame callback.
Sources: apps/web/src/lib/live/useLiveSession.ts:252-277
Persistence
The system uses a JSON-file store for settings and "remembered" notes. The remember tool appends facts to an array in the agent_notes setting, which is then injected into the system prompt for subsequent interactions.
Sources: services/agent/src/tools.ts:138-155, services/agent/src/prompt.ts:63-68
Summary
Nudge is a specialized tutoring system that leverages local AI models to provide non-intrusive, Socratic support. By combining a floating "Sphere" interface with proactive screen observation and a tiered intervention policy, it aims to keep students in their flow state while providing the necessary "nudges" to overcome learning bottlenecks.
Sources: website/src/renderer.js:14-20, README.md:3-6