Dual Ollama Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Dual Ollama Strategy is a specialized architectural pattern implemented in the Nudge project to optimize performance and latency in a live voice tutoring environment. By bifurcating the responsibilities of the artificial intelligence into two distinct roles—"Talk" and "Eyes"—the system achieves fast spoken replies while maintaining continuous, high-fidelity visual context. This approach specifically addresses the latency bottlenecks associated with sending large image prefills to language models by using separate, specialized local models for vision and speech.
Sources: README.md:9-25, README.md:50-58
Core Architecture and Roles
The system designates specific local models for different tasks to ensure that spoken turns do not re-send full screen frames to the vision model every time, which would significantly increase time-to-first-word.
Model Roles and Specifications
| Role | Default Local Model | Primary Responsibility | Technical Constraint |
|---|---|---|---|
| Talk | gemma4:e2b-it-qat | Fast spoken replies and natural conversation. | Minimal/No image prefill to ensure sub-1s latency. |
| Eyes | qwen2.5vl:7b | Screen observation and vision-to-text summarization. | Optimized for high-fidelity OCR and visual scene analysis. |
Sources: README.md:12-19, README.md:50-58, services/agent/src/live/turn-runner.ts:8-10
Logic and Data Flow
The "Eyes" model operates on a proactive observation loop, capturing frames from the user's screen or camera and generating "Ground Truth" summaries. These summaries are then injected as text into the "Talk" model's prompt. This prevents the "Talk" model from needing to process raw image data directly, which is the primary source of latency in multimodal LLM interactions.
Vision-Speech Separation
The logic for determining which model performs vision tasks is handled by the resolveEyes method within the LiveTurnRunner.
Vision Resolution Logic
The system prefers a dedicated vision model if configured in settings. If no dedicated vision model is present, it attempts to use the primary "Talk" model only if that model supports vision capabilities (e.g., vl, llava, qwen2.5vl).
Sources: services/agent/src/live/turn-runner.ts:133-143, apps/web/src/components/settings/ModelsSettings.tsx:120-140
Proactive Observation Loop
The useLiveSession hook orchestrates an observeTimer that periodically triggers frame capture. The tickObserve function ensures that observation does not occur while the student is talking or the tutor is mid-reply to avoid context interference.
Implementation Details
Frame Summarization
The "Eyes" model is prompted to provide a dense, factual, one-line summary of on-screen text, equations, and student actions. This summary is constrained by OBSERVE_MAX_TOKENS (default 120) to maintain speed.
// From services/agent/src/live/turn-runner.ts:69-75
const OBSERVE_VISION_PROMPT = `Look at the student's screen right now.
Reply in EXACTLY this two-line format:
SUMMARY: <one dense factual line: problem type + key on-screen text/numbers/equations (quote briefly if readable) + what the student appears to be doing>
SPEAK: SILENCE
or
SUMMARY: <one dense factual line as above>
SPEAK: <one short Socratic hint or question, one sentence>`;
Sources: services/agent/src/live/turn-runner.ts:69-77, services/agent/src/live/turn-runner.ts:172-180
The Delegate Tool
The strategy extends to web searching and fact-checking via the delegate tool. The main "Talk" agent hands off complex research tasks to a "Worker" subagent. The worker handles the multi-step "tool grind" (web searching, URL fetching) so the main agent's context remains small and response times stay low.
Sources: services/agent/src/tools.ts:153-165, services/agent/src/live/worker.ts:47-60
Configuration and Constraints
The Dual Ollama Strategy utilizes specific parameters to manage model behavior and resource consumption.
| Parameter | Value / File Source | Description |
|---|---|---|
TALK_MAX_TOKENS | 384 (turn-runner.ts:7) | Limits the length of the spoken response. |
OBSERVE_MAX_TOKENS | 120 (turn-runner.ts:8) | Limits the length of the vision summary. |
MAX_STEPS | 6 (turn-runner.ts:65) | Maximum tool rounds allowed before the model must speak. |
NARRATE_AFTER_MS | 1500 (worker.ts:14) | Time threshold before the worker provides a progress narration. |
Sources: services/agent/src/live/turn-runner.ts:7-65, services/agent/src/live/worker.ts:14-16
Summary
The Dual Ollama Strategy is essential for the Nudge "quiet tutor" experience. By offloading visual processing to a dedicated model loop and feeding textual summaries to a fast conversation model, the project overcomes the latency issues typical of local LLM vision processing. This architecture ensures that the tutor remains "context-aware" through the Eyes model while remaining "calm and precise" in its spoken interactions via the Talk model.
Sources: README.md:1-5, README.md:50-58