LLM Providers & Routing
Relevant source files
The following files were used as context for generating this wiki page:
The LLM Providers & Routing system within tldw_server serves as a unified abstraction layer for interacting with over 16 different Large Language Model (LLM) backends, ranging from commercial APIs to local inference engines. This architecture ensures that the rest of the application—including Chat and RAG modules—can interact with diverse models through a consistent, OpenAI-compatible interface while abstracting away provider-specific API nuances, rate limits, and connection handling.
Sources: Docs/Architecture.md:43-46, CLAUDE.md:65-69, AGENTS.md:68-72
Provider Architecture
The project utilizes a pluggable provider layer located in tldw_Server_API/app/core/LLM_Calls/. This layer acts as the central hub for adding new providers or tweaking existing ones. It handles API key management, streaming orchestration, and error normalization.
Supported Provider Types
The system categorizes providers into two main types:
| Category | Supported Providers |
|---|---|
| Commercial | OpenAI, Anthropic, Cohere, DeepSeek, Google, Groq, HuggingFace, Mistral, OpenRouter, Qwen, Moonshot, Z.AI |
| Local / Self-Hosted | Llama.cpp, Kobold.cpp, Oobabooga, TabbyAPI, vLLM, Ollama, Aphrodite, Custom OpenAI-compatible |
Sources: CLAUDE.md:65-69, AGENTS.md:68-72
Managed Inference Instances
For local providers like vLLM, the system implements a "Managed Instances" control plane. This allows for persistent storage of instance configurations, lifecycle management (start/stop/restart), and request-scoped routing. This managed approach supports both local execution and execution via SSH on remote GPU hardware.
Sources: Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:10-15
Request Flow and Routing Logic
Routing in tldw_server is typically request-scoped, meaning the specific provider and model are resolved at the time an API call is made to the /chat/completions or /embeddings endpoints.
High-Level Request Flow
The following diagram illustrates how a chat request moves from a client through the routing layer to a specific LLM provider.
The endpoint calls into core modules like core/Chat/ or core/LLM_Calls/, which handle provider routing and streaming orchestration.
Sources: Docs/Architecture.md:73-86, Docs/Architecture.md:183-195
Routing Resolution
For managed instances (specifically vLLM), a dedicated resolver.py manages instance selection. The resolution logic prioritizes:
- Explicit Instance ID: A
provider_instance_idprovided in the request payload. - Default Route: The globally configured default instance for that provider type.
- Capabilities: Verification that the resolved instance supports the required capability (e.g., "chat", "embeddings", or "vision").
Sources: Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:78-95
Key Components and Data Structures
Core Modules
- LLM_Calls (
app/core/LLM_Calls/): The provider abstraction layer for 16+ backends. It handles rate limits, error handling, and streaming. - Chat (
app/core/Chat/): Manages OpenAI-compatible completions, history management, and character cards. - VLLM_Management (
app/core/VLLM_Management/): A control plane for managing the lifecycle and routing of vLLM instances.
Sources: Docs/Architecture.md:126-135, Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:10-15
Managed Instance Schema
The system uses a structured record to manage model instances, particularly for local inference.
| Field | Type | Description |
|---|---|---|
instance_id | str | Unique identifier for the instance. |
execution_mode | str | local or ssh. |
transport_config | dict | Connection details (host, port, auth, launcher path). |
launch_spec | dict | Model parameters (model name, port, tensor parallelism). |
routing_policy | dict | Logic for including the instance in automatic routing. |
declared_capabilities | dict | Supported features (chat, embeddings, vision). |
Sources: Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:46-55
Sequence: Managed Instance Lifecycle
This sequence diagram shows the asynchronous orchestration of a managed LLM instance using the Jobs system.
Lifecycle actions like start, stop, and restart are processed as background jobs to prevent blocking the API.
Sources: Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:200-225
Configuration and Secrets
Configuration for LLM providers is managed through a combination of Config_Files/config.txt and environment variables in .env. Sensitive information, such as API keys, is preferred in .env and is never logged or echoed back in API responses. For managed SSH instances, secrets are handled via write-only encrypted references to ensure credentials are not exposed in plaintext.
Sources: CLAUDE.md:139-145, AGENTS.md:144-150, Docs/Plans/2026-03-10-vllm-managed-instances-implementation-plan.md:58-60
Summary
LLM Providers & Routing provides the technical backbone for the intelligence features of tldw_server. By abstracting multiple commercial and local backends into a single OpenAI-compatible interface, the system allows for flexible model switching and sophisticated resource management. The introduction of managed vLLM instances further extends this by providing a durable control plane for local and remote GPU-based inference.
Sources: Docs/Architecture.md:183-195, CLAUDE.md:31-35