Error Handling and Resilience
Relevant source files
The following files were used as context for generating this wiki page:Loom is designed as a runtime for long-lifecycle agents, making error handling and resilience critical features. The system's approach is multi-layered, encompassing everything from low-level tool execution in the Rust Core to high-level agent process management in the Python SDK. This ensures that agents can operate autonomously, recover from failures, and provide clear feedback when issues arise.
The architecture separates concerns, with the Rust Core handling robust tool and protocol errors, while the Python layer manages process lifecycle, network communication failures, and user-facing error presentation. This layered strategy contributes to the overall stability and reliability of the agent ecosystem.
Rust Core Error Handling
The Rust Core forms the foundation of Loom's resilience, providing structured error types for tool execution and inter-process communication.
Tool Execution Errors
The primary error type for all tool operations is the ToolError enum. This provides a standardized way to represent failures across native tools, MCP tools, and the tool registry itself.
Sources: core/src/tools/error.rs
The diagram below shows the different variants of the ToolError enum.
The table below details each variant of the ToolError enum.
| Variant | Description |
|---|---|
InvalidInput | Indicates that the parameters provided to a tool were invalid. |
ExecutionFailed(String) | A generic error for when a tool fails during its execution. |
ToolNotFound(String) | Occurs when a requested tool is not found in the ToolRegistry. |
IoError(String) | Wraps standard I/O errors, often from filesystem operations. |
McpError(String) | Represents an error originating from a Model Context Protocol (MCP) server. |
Other(String) | A catch-all for any other type of error. |
Sources: core/src/tools/error.rs:8-21
Tool Orchestrator Error Flow
The ToolOrchestrator is responsible for managing LLM interactions and tool calls. When a tool execution fails, the orchestrator catches the ToolError and packages it into a structured ActionResult protobuf message. This ensures that failures are communicated cleanly back through the system.
The following sequence diagram illustrates the error handling flow within the ToolOrchestrator when a tool call fails.
This process is implemented in the run method of the ToolOrchestrator. If self.tools.call(...) returns an Err, it's converted into an ActionResult with status: ActionStatus::ActionError.
// core/src/cognitive/llm/tool_orchestrator.rs:202-218
let result = self.tools.call(&call.name, call.arguments.clone()).await;
let res = match result {
Ok(val) => ActionResult {
// ... success case
},
Err(e) => ActionResult {
id: call.id.clone().unwrap_or_default(),
status: ActionStatus::ActionError as i32,
output: Vec::new(),
error: Some(crate::proto::ActionError {
code: "500".to_string(),
message: e.to_string(),
details: std::collections::HashMap::new(),
}),
},
};
Sources: core/src/cognitive/llm/tool_orchestrator.rs:199-220
LLM Client Resilience
The ToolOrchestrator also includes resilience for LLM API calls. It first attempts to use the /responses API endpoint. If that request fails (e.g., due to an unsupported endpoint or other HTTP error), it gracefully falls back to trying the standard /chat/completions endpoint. This fallback mechanism increases compatibility with different OpenAI-compatible servers.
Sources: core/src/cognitive/llm/tool_orchestrator.rs:151-169
MCP Error Types
For tools integrated via the Model Context Protocol (MCP), there are specific error codes tested in the codebase that map to McpError variants.
| MCP Error Code | Description |
|---|---|
TIMEOUT | The MCP tool call timed out. |
TOOL_NOT_FOUND | The requested tool was not found on the MCP server. |
INVALID_PARAMS | The parameters sent to the MCP tool were invalid. |
Sources: core/tests/mcp_test.rs:205-219
Python SDK Resilience
The Python SDK (loom-py) builds on the Core's foundation with features for managing network failures, process lifecycle, and user interaction.
Bridge Communication Failures
The BridgeClient handles gRPC communication with the Rust Core. The SDK anticipates connection failures and uses a standard try...except RpcError block to handle them, allowing agent code to implement retry logic or other fallback behaviors.
The sequence diagram below shows how a Python agent would handle a gRPC connection error.
The loom-py documentation explicitly provides an example for handling this.
from grpc import RpcError
try:
await client.connect()
except RpcError as e:
print(f"Failed to connect: {e.code()} - {e.details()}")
# Retry logic...
Sources: loom-py/src/loom/bridge/README.md
Agent Process Orchestration
The loom CLI and runtime provide an Orchestrator that manages the lifecycle of both the core runtime and the Python agent processes. Key resilience features include:
- Restart on Failure: The orchestrator automatically restarts agent processes if they crash.
- Graceful Shutdown: When
loom downis executed, it sends aSIGTERMsignal to all processes, allowing them to shut down gracefully. If they do not terminate within a 3-second timeout, it escalates toSIGKILLto ensure a clean state.
The following flowchart illustrates the graceful shutdown process.
Security and Error Prevention
Resilience is also achieved by preventing errors, particularly those related to security. The filesystem tools are a key example.
- Workspace Isolation: Filesystem tools are restricted to a designated workspace root directory.
- Path Traversal Blocking: The tools actively block attempts to use paths like
../to access files outside the workspace. This is validated by unit tests.
// core/tests/filesystem_tool_test.rs:118-129
#[tokio::test]
async fn test_path_traversal_blocked() {
let workspace = create_temp_workspace();
let write_tool = WriteFileTool::new(workspace.path().to_path_buf());
// Try to write outside workspace
let result = write_tool
.call(json!({
"path": "../../../etc/passwd",
"content": "hacked"
}))
.await;
assert!(result.is_err());
}
Sources: core/tests/filesystem_tool_test.rs:118-129, core/src/tools/README.md
User-Facing Error Presentation
Errors that bubble up to the user are handled by the CLI's UI module, which provides formatted and styled output to make them easily understandable.
The loom.cli.ui module contains functions like print_error, print_warning, and print_connection_status to present system state and failures clearly in the terminal.
Sources: loom-py/src/loom/cli/ui.py:10, 139-148
Real-World Troubleshooting
The voice_agent demo's documentation provides a practical list of common runtime errors and their resolutions, demonstrating how the system's error handling manifests in a real application.
| Scenario | Cause & Resolution |
|---|---|
| Microphone not found | The specified audio device is unavailable. Resolved by listing devices and setting the MIC_DEVICE environment variable. |
| No STT (Speech-to-Text) | The whisper.cpp binary or model is missing. Resolved by ensuring paths in WHISPER_BIN and WHISPER_MODEL_PATH are correct. |
| LLM errors | The LLM backend is unavailable or misconfigured. Resolved by checking the VLLM_BASE_URL and ensuring the server is running. |
| No audio playback | A required playback utility (aplay, paplay) is not installed. |
| TTS "shared library not found" | The Piper TTS binary was installed without its required dependencies. Resolved by installing the complete release package. |
Sources: apps/voice_agent/README.md
Conclusion
Loom's approach to error handling and resilience is comprehensive, addressing potential failures at every level of the stack. From structured ToolError types in the Rust Core and fallback logic in the ToolOrchestrator, to process lifecycle management and graceful shutdowns in the Python SDK, the system is built for stability. This robust foundation enables developers to build complex, long-running AI agents that can recover from transient errors and operate reliably in dynamic environments.