Agent Turn Runner
Relevant source files
The following files were used as context for generating this wiki page:
The Agent Turn Runner is a core back-end system responsible for orchestrating the execution of conversational turns between a user and an AI agent. It manages the lifecycle of a "turn," which includes processing user text input, interpreting visual data from camera or screen frames, executing tools (such as web searches or memory updates), and maintaining conversation history. Within the OpenLive architecture, it serves as the bridge between the live WebSocket connection and the underlying Large Language Model (LLM) providers.
The system is designed for high-performance live interactions, employing strategies like prompt cache warming to minimize first-token latency and visual pre-processing to handle multi-modal inputs. It supports a "Thick Client" architecture where heavy voice processing occurs on the client device, while the Turn Runner handles the semantic reasoning and tool execution logic on the server.
Sources: services/agent/src/live/turn-runner.ts:98-102, packages/shared/src/live-events.ts:11-17
Core Architecture and Components
The architecture centers around the LiveTurnRunner class, which maintains the state of a live session, including message history and system prompts.
Key Components
| Component | Description | Source |
|---|---|---|
| LiveTurnRunner | Main class orchestrating turns, managing message history, and handling multi-modal inputs. | services/agent/src/live/turn-runner.ts:98 |
| Worker Subagent | A delegated agent that executes complex, multi-step tool loops (e.g., web research) to keep the main conversation responsive. | services/agent/src/live/worker.ts:40 |
| Vision Processor | Functions like describeFrames that convert visual input (camera/screen) into textual summaries for the LLM. | services/agent/src/live/turn-runner.ts:18 |
| Tool System | A registry of executable functions (TaktTools) that the agent can invoke, such as web_search, remember, and delegate. | services/agent/src/tools.ts:15 |
System Data Flow
The following diagram illustrates the lifecycle of a single user turn within the Turn Runner:
Execution Logic
Turn Processing (runTurn)
The runTurn method is the primary entry point for processing user interactions. It follows a structured sequence:
- Provider Resolution: Identifies the configured LLM provider and API keys.
- Multi-modal Handling: If frames are provided, it either attaches them directly to the user message or uses a separate vision model to describe them via
describeScreento save context space. - History Management: Appends the user message to the local history, ensuring that old frames are purged to minimize latency and cost.
- Inference Loop: Starts a loop (capped at 6 steps) to handle reasoning and tool execution.
- Tool Execution: If the LLM generates a tool call, the runner executes the tool and appends the result back to the message history for the next iteration.
Sources: services/agent/src/live/turn-runner.ts:162-230
Vision and Observation
For "Study Tutor" mode, the system supports proactive observation. The runObserveVision method uses a dedicated vision model to peek at the student's screen and decide whether to speak or remain silent based on progress.
Tooling and Delegation
The Turn Runner utilizes a delegation model for complex tasks. While the main agent handles the voice conversation, it can delegate research tasks to a Worker Subagent.
Worker Subagent Logic
The Worker Subagent runs in a separate loop (runWorker) and has access to "web tools" like web_search and fetch_url. Its output is purely textual findings which the main agent then relays to the user. To prevent "dead air" during long research tasks, the worker can emit narrations (e.g., "Still searching for...") while it works.
Sources: services/agent/src/live/worker.ts:40-74, services/agent/src/tools.ts:114-127
TaktTools Registry
Tools are defined by the TaktTool interface, requiring a name, description, parameters (Zod schema), and an execution function.
| Tool Name | Scope | Description |
|---|---|---|
web_search | Worker | Searches the web via Exa API. |
fetch_url | Worker | Fetches and cleans HTML from a specific URL. |
delegate | Main | Hands off a research task to the worker subagent. |
remember | Main | Persists facts in agent_notes across sessions. |
update_todos | Main | Updates a visual checklist in the UI. |
| Sources: services/agent/src/tools.ts:15-127 |
Performance Optimizations
Prompt Cache Warming
To reduce the latency of the first token in a session, the Turn Runner includes a warm method. This sends a tiny, 1-token request containing the system prompt and tool definitions to the provider immediately upon session opening. This triggers "Prompt Caching" (specifically for providers like Anthropic), making the first actual user turn a cache hit.
Sources: services/agent/src/live/turn-runner.ts:112-124, packages/harness/src/anthropic.ts:48-58
History Capping
To prevent unbounded growth of the context window, the runner implement capHistory. It truncates the message history once it exceeds a threshold (40 messages), ensuring the cut happens at a user boundary to avoid orphaning tool calls.
Sources: services/agent/src/live/turn-runner.ts:129-134
Summary
The Agent Turn Runner provides the orchestration layer for live, multi-modal AI interactions. By separating concerns between a responsive main agent and a background worker subagent, and by aggressively optimizing for latency through prompt warming and history management, it enables a seamless "Study Tutor" experience that can see, hear, and research in real-time.
Sources: services/agent/src/live/turn-runner.ts, services/agent/src/prompt.ts