Interactive Runner
Relevant source files
The following files were used as context for generating this wiki page:
The Interactive Runner is a specialized testing component within the Osmosis SDK designed for step-by-step agent loop evaluation. It allows developers to pause execution after each LLM call to inspect conversation messages, tool invocations, and intermediate state transitions. This mode is particularly useful for debugging complex agentic workflows where multi-turn reasoning and tool use must be verified manually.
The system operates by wrapping standard LLM clients with an interactive layer that intercepts completions and prompts for user feedback. It integrates directly with the osmosis test CLI command via the --interactive flag, enabling a seamless transition from batch testing to targeted, interactive debugging of specific dataset rows.
Sources: osmosis_ai/rollout/test_mode/interactive.py:1-12, osmosis_ai/rollout/test_mode/cli.py:13-20
Architecture and Components
The Interactive Runner architecture is built around a wrapper pattern where the InteractiveLLMClient proxies calls to an underlying ExternalLLMClient. This allows the runner to maintain the standard LLMClientProtocol while introducing human-in-the-loop control.
Core Classes
| Class | Description |
|---|---|
InteractiveRunner | Main orchestrator that manages the session, handles user input, and prints formatted output to the console. |
InteractiveLLMClient | A wrapper for ExternalLLMClient that triggers callbacks before and after LLM completions. |
InteractiveStep | A dataclass representing a discrete unit of execution (e.g., an LLM response or tool result). |
Sources: osmosis_ai/rollout/test_mode/interactive.py:23-38, osmosis_ai/rollout/test_mode/interactive.py:114-125
Component Interaction Diagram
The following diagram illustrates how the InteractiveRunner sits between the Agent Loop and the external LLM provider to facilitate step-by-step execution.
Execution Flow and Session Management
The interactive session starts by loading a dataset and initializing the agent loop. Unlike batch mode, which processes rows sequentially without interruption, the interactive session provides a selection menu for the user to pick specific absolute row indices from the dataset.
Session Initialization
The run_interactive_session method manages the top-level loop, allowing users to select rows based on absolute indices (accounting for CLI --offset and --limit parameters).
Step-by-Step Logic
During an active agent run, every call to chat_completions triggers a state update and a block.
- Message Tracking: The
InteractiveLLMClientcallson_messages_updatedso the runner can mirror the current conversation state for the user. - LLM Invocation: The actual call to the provider is made via the wrapped client.
- User Prompt: An
InteractiveStepobject is created, containing the turn number, message content, and any detected tool calls. The execution pauses until the user provides a command.
Sources: osmosis_ai/rollout/test_mode/interactive.py:75-103, osmosis_ai/rollout/test_mode/interactive.py:415-434
Command Interface
The runner provides a set of console-based commands to control the flow of the agent.
| Command | Aliases | Action |
|---|---|---|
next | n, "" (empty) | Proceeds to the very next step in the agent loop. |
continue | c | Disables further pauses and runs the current row to completion. |
messages | m | Pretty-prints the entire conversation history up to the current turn. |
tools | t | Displays all tools currently available to the agent. |
quit | q, exit | Aborts the current execution and returns to the row selection menu. |
Sources: osmosis_ai/rollout/test_mode/interactive.py:127-143, osmosis_ai/rollout/test_mode/interactive.py:344-385
Implementation Details
Interactive LLM Client Wrapping
The InteractiveLLMClient is injected into the RolloutContext in place of the standard client. It ensures that any agent framework using the context's LLM capabilities is automatically subject to the interactive breakpoints.
# osmosis_ai/rollout/test_mode/interactive.py:433-441
# Create interactive client wrapper
interactive_client = InteractiveLLMClient(
wrapped_client=self.llm_client,
on_step=self._handle_step,
on_messages_updated=update_messages,
)
interactive_client.set_tools(tools)
# Create context with interactive client
ctx = RolloutContext(
request=request,
tools=tools,
llm=interactive_client, # type: ignore
_start_time=agent_start_time,
_debug_dir="./test_debug" if self.debug else None,
)
Sources: osmosis_ai/rollout/test_mode/interactive.py:433-441, tests/test_test_mode_runner.py:347-360
Tool Validation
In interactive mode, tools are validated via validate_tools before the agent loop starts. This ensures that the runner does not attempt to display or invoke malformed tool schemas, providing immediate feedback on schema errors such as missing function names or invalid parameter types.
Sources: osmosis_ai/rollout/test_mode/interactive.py:421-423, osmosis_ai/rollout/test_mode/runner.py:27-38
Conclusion
The Interactive Runner serves as a critical bridge between automated batch testing and manual debugging. By providing a human-in-the-loop interface for the RolloutAgentLoop, it enables fine-grained inspection of LLM interactions and tool executions. This module ensures that developers can verify not just the final output (reward) of an agent, but the validity of its internal reasoning chain.