LLM Provider Integration
Relevant source files
The following files were used as context for generating this wiki page: - [loom-py/src/loom/llm/provider.py](https://github.com/Alonso-pixel/loom/blob/HEAD/loom-py/src/loom/llm/provider.py) - [apps/market-analyst/loom.toml](https://github.com/Alonso-pixel/loom/blob/HEAD/apps/market-analyst/loom.toml) - [loom-py/src/loom/llm/README.md](https://github.com/Alonso-pixel/loom/blob/HEAD/loom-py/src/loom/llm/README.md) - [ROADMAP.md](https://github.com/Alonso-pixel/loom/blob/HEAD/ROADMAP.md) - [apps/voice_agent/README.md](https://github.com/Alonso-pixel/loom/blob/HEAD/apps/voice_agent/README.md) - [core/src/cognitive/llm/provider.rs](https://github.com/Alonso-pixel/loom/blob/HEAD/core/src/cognitive/llm/provider.rs) - [core/tests/llm_test.rs](https://github.com/Alonso-pixel/loom/blob/HEAD/core/tests/llm_test.rs) - [core/src/cognitive/README.md](https://github.com/Alonso-pixel/loom/blob/HEAD/core/src/cognitive/README.md)Loom's architecture for Large Language Model (LLM) integration is centered around a "Brain/Hand Separation" principle. The "Brain," implemented in Python, handles cognitive tasks like reasoning and prompt engineering, and makes direct HTTP calls to LLM APIs for fast iteration. The "Hand," implemented in Rust, manages stable infrastructure like tool execution and the event bus, but does not handle LLM calls for Python agents. This separation allows developers to rapidly modify prompts and LLM configurations in Python without recompiling the Rust core.
This document details how LLM providers are configured and utilized in both the Python SDK and the Rust core, reflecting the project's dual-language approach to building AI agents.
Sources: ROADMAP.md, loom-py/src/loom/llm/README.md:5-9
Architecture
The core architectural decision is to keep LLM interactions within the Python domain for agents, promoting flexibility and rapid development.
This diagram shows the "Brain/Hand" separation. The Python CognitiveAgent uses LLMProvider to call external LLM APIs directly, while interacting with the Rust Core for sandboxed tool execution.
Sources: ROADMAP.md, loom-py/src/loom/llm/provider.py:5-9
While Python agents use the direct HTTP approach, the Rust core also contains its own LlmClient for internal use or for agents/demos written purely in Rust, such as the Voice Agent. This provides a complete, self-contained LLM capability within the core runtime when needed.
Sources: core/src/cognitive/README.md:10-16, apps/voice_agent/README.md
Python LLM Integration (LLMProvider)
The primary mechanism for LLM interaction in Loom is the LLMProvider class within the Python SDK. It acts as a direct HTTP client to various LLM APIs.
Sources: loom-py/src/loom/llm/provider.py, loom-py/src/loom/llm/README.md
Key Components
| Class/Component | Description | File Path |
|---|---|---|
LLMProvider | The main class for making synchronous and streaming calls to LLM APIs. It manages configuration, authentication, and HTTP requests. | loom-py/src/loom/llm/provider.py:24-325 |
LLMConfig | A data structure holding the configuration for an LLM connection, including base URL, model name, API key, and other parameters. | loom-py/src/loom/llm/provider.py:33-55 |
| OpenTelemetry Tracer | All LLM calls are automatically traced, creating spans with attributes like llm.provider, llm.model, and token usage. | loom-py/src/loom/llm/provider.py:17, 131-145 |
Instantiation
An LLMProvider can be created in two main ways:
-
From a pre-configured name: The class includes presets for common providers.
# Create a provider for DeepSeek, OpenAI, or a local model llm = LLMProvider.from_name(agent.ctx, "deepseek") -
From project configuration: It can be loaded dynamically from a
loom.tomlfile. This is the recommended approach for projects.# In your agent code from loom.runtime import load_project_config project_config = load_project_config("loom.toml") llm = LLMProvider.from_config(agent.ctx, "deepseek", project_config)
Pre-configured Providers
The LLMProvider comes with built-in configurations for popular services.
| Provider Name | Default Model | Default Base URL | API Key Environment Variable |
|---|---|---|---|
deepseek | deepseek-chat | https://api.deepseek.com/v1 | DEEPSEEK_API_KEY |
openai | gpt-4o-mini | https://api.openai.com/v1 | OPENAI_API_KEY |
local | qwen2.5-0.5b-instruct | http://localhost:8000/v1 | None |
Sources: loom-py/src/loom/llm/provider.py:33-55, loom-py/src/loom/llm/README.md:30-48
Request Flow
The following diagram shows the sequence of a typical LLM call from a CognitiveAgent.
This sequence illustrates a call to the generate method, including the creation of an OpenTelemetry span to trace the operation.
Sources: loom-py/src/loom/llm/provider.py:131-203
Rust LLM Integration (LlmClient)
The Rust core includes its own LlmClient for use cases that operate entirely within the Rust ecosystem, like the voice_agent demo.
Sources: core/src/cognitive/README.md:10-16, core/tests/llm_test.rs
Configuration
The Rust LlmClient is configured primarily through environment variables, which are loaded into an LlmClientConfig struct.
| Environment Variable | Default Value | Description |
|---|---|---|
VLLM_BASE_URL | http://localhost:8000/v1 | The base URL for the OpenAI-compatible API. |
VLLM_MODEL | qwen2.5-0.5b-instruct | The model name to use. |
VLLM_API_KEY | (none) | The API key for authentication. |
REQUEST_TIMEOUT_MS | 30000 | The request timeout in milliseconds. |
VLLM_TEMPERATURE | 0.7 | The sampling temperature. |
This configuration is demonstrated in the voice_agent demo and verified in unit tests.
Sources: core/tests/llm_test.rs:6-23, 27-44, apps/voice_agent/README.md:100-106
Configuration (loom.toml)
LLM providers are defined in the loom.toml file, allowing for centralized management of models, endpoints, and credentials.
Example Configuration
This example from the market-analyst application defines a provider named deepseek.
# apps/market-analyst/loom.toml
[llm.deepseek]
type = "http"
api_key = "${DEEPSEEK_API_KEY}"
api_base = "https://api.deepseek.com"
model = "deepseek-chat"
max_tokens = 4096
temperature = 0.7
The Python LLMProvider loads this configuration, resolving the api_key from the DEEPSEEK_API_KEY environment variable.
Sources: apps/market-analyst/loom.toml:19-26, loom-py/src/loom/llm/provider.py:108-121
A similar structure is used for configuring the Rust client in the voice_agent demo.
# apps/voice_agent/voice_agent.toml (example)
[llm]
base_url = "http://localhost:8000/v1"
model = "Qwen/Qwen2.5-0.5B-Instruct"
temperature = 0.6
request_timeout_ms = 30000
system_prompt = "You are Loom's helpful and concise voice assistant."
Sources: apps/voice_agent/README.md:51-57
Configuration Loading Flow
This diagram shows how both Python and Rust components source their LLM configurations from loom.toml and environment variables.
Sources: loom-py/src/loom/llm/provider.py:87-130, core/tests/llm_test.rs:27-38
Conclusion
Loom's LLM integration strategy provides significant flexibility. By prioritizing direct HTTP calls from the Python "Brain," it empowers developers to rapidly prototype and refine agent behavior. Simultaneously, the inclusion of an LlmClient in the Rust "Hand" ensures that the core runtime and Rust-native applications have access to LLM capabilities when needed. This dual approach, managed through a unified loom.toml configuration, supports a wide range of agent architectures and development workflows.
Sources: ROADMAP.md, loom-py/src/loom/llm/README.md