LLM Provider System

Relevant source files

The following files were used as context for generating this wiki page:

• src/providers/LLMProvider.ts
• src/providers/MLXProvider.ts
• src/providers/LlamaCppProvider.ts
• src/core/agent.ts
• prd/multi_provider_support.md

The LLM Provider System is a modular architecture within the Autohand CLI designed to abstract interactions with various Large Language Model (LLM) backends. It supports a diverse range of providers including cloud-based services like OpenRouter and OpenAI, as well as local inference engines such as Ollama, llama.cpp, and MLX. This abstraction allows the core agent logic to remain provider-agnostic while enabling users to switch between models and backends seamlessly through configuration or runtime commands.

Sources: prd/multi_provider_support.md:5-10, src/core/agent.ts:133-145

System Architecture

The architecture follows a provider pattern where a unified interface defines the contract for LLM interactions. A factory class is responsible for instantiating the appropriate provider implementation based on the user's configuration.

Provider Interface

Every backend must implement the LLMProvider interface, ensuring consistent behavior for model completion, availability checks, and model listing.

The diagram above illustrates the relationship between the base provider interface and specific implementations.
Sources: prd/multi_provider_support.md:58-65, src/providers/MLXProvider.ts:41-45, src/providers/LlamaCppProvider.ts:38-42

Data Flow: Request and Response

The system uses standardized LLMRequest and LLMResponse structures to handle chat completions. Requests include message history, temperature, and tool definitions, while responses provide content, tool calls, and token usage metrics.

Component	Description	Fields
LLMRequest	Input sent to the provider	messages, model, temperature, maxTokens, tools, signal
LLMResponse	Output from the provider	id, content, toolCalls, finishReason, usage
LLMUsage	Token consumption data	promptTokens, completionTokens, totalTokens

Sources: src/providers/MLXProvider.ts:86-115, src/providers/LlamaCppProvider.ts:74-101

Supported Providers

The system categorizes providers into Cloud and Local types, each with specific configuration requirements and optimization paths.

Local Providers

1. MLX: Specialized for Apple Silicon (M1/M2/M3) using mlx-lm. It utilizes the OpenAI-compatible /v1/chat/completions endpoint and is only available on supported macOS platforms.
2. llama.cpp: A high-performance local inference server. It typically operates on port 8080 and provides a /health endpoint for availability checks.
3. Ollama: A local runner that supports automatic model fetching via the /api/tags endpoint.

Sources: prd/multi_provider_support.md:21-36, src/providers/MLXProvider.ts:46-75, src/providers/LlamaCppProvider.ts:63-72

Cloud Providers

1. OpenRouter: A unified API aggregator providing access to diverse models like Claude 3.5 Sonnet.
2. OpenAI: Native support for GPT-4o, GPT-4 Turbo, and GPT-3.5 models.

Sources: prd/multi_provider_support.md:15-20, src/core/agent.ts:544-585

Provider Implementation Logic

Providers translate internal LLMRequest objects into the specific JSON payloads required by their respective REST APIs.

The sequence shows how the agent interacts with providers through the factory and abstraction layer.
Sources: src/core/agent.ts:162-175, src/providers/MLXProvider.ts:81-125, src/providers/LlamaCppProvider.ts:74-115

Tool Calling Integration

Providers check for the presence of tools in the LLMRequest. If tools are provided, they are mapped to the provider's specific function calling format (usually type: 'function'). Upon receiving a response, the provider parses tool_calls and returns them as part of the LLMResponse with a finishReason of tool_calls.

Sources: src/providers/MLXProvider.ts:101-115, src/providers/LlamaCppProvider.ts:89-105

Configuration and Selection

Configuration is persisted in a config.json file, allowing users to define multiple provider profiles.

Configuration Schema Example

{
  "provider": "ollama",
  "openrouter": {
    "apiKey": "sk-or-...",
    "model": "anthropic/claude-3.5-sonnet"
  },
  "mlx": {
    "baseUrl": "http://localhost:8080",
    "model": "mlx-community/Llama-3.2-3B-Instruct-4bit"
  }
}

Sources: prd/multi_provider_support.md:43-56, src/core/agent.ts:505-520

Runtime Model Switching

The AutohandAgent manages provider state through methods like applyModelChange. Switching a provider involves recreating the LLM client through the factory, updating the context window settings, and potentially re-initializing delegators if sub-agents are in use.

Sources: src/core/agent.ts:684-706, src/core/agent.ts:1445-1466

Summary

The LLM Provider System provides a robust foundation for multi-backend support, ensuring the Autohand CLI can leverage both the power of cloud models and the privacy/speed of local inference. By standardizing interfaces and request/response cycles, the system simplifies the integration of new LLM technologies as they emerge.