LLM Provider Integrations (LiteLLM)
Relevant source files
The following files were used as context for generating this wiki page:
Onyx utilizes LiteLLM as a unified interface to support a wide array of Large Language Model (LLM) providers. This integration allows the platform to interact with different models—such as those from OpenAI, Anthropic, Bedrock, and local instances like Ollama or LM Studio—using a consistent API. By abstracting provider-specific complexities, LiteLLM enables features like multi-model support for chat, search, and embeddings while maintaining specialized logic for individual model quirks like "reasoning" efforts or tool-calling formats.
Sources: AGENTS.md:162-166, backend/onyx/llm/multi_llm.py:195-200
Architecture and Core Components
The LLM integration is built around the LitellmLLM class, which implements the LLM interface. This class handles the translation of Onyx's internal prompt structures into LiteLLM-compatible dictionaries and manages the complexities of provider-specific configurations.
Request Flow and Execution
The system processes LLM interactions in steps or cycles. A single inference (step) is wrapped by functions that handle packet translation, ensuring the UI can emit tokens (reasoning or answers) as soon as they arrive from the stream.
The diagram shows the data flow from a user request through the backend processing layers to the final token emission via LiteLLM.
Sources: backend/onyx/chat/README.md:144-165, backend/onyx/llm/multi_llm.py:270-300
Key Classes and Data Structures
| Component | Description |
|---|---|
LitellmLLM | Main implementation class that interfaces with the LiteLLM library. |
LanguageModelInput | A stripped-down representation of a message used specifically for the LLM interface layer. |
ModelResponse | Standardized object containing the final output, usage statistics, and cost information. |
LLMConfig | Stores provider name, model name, temperature, API keys, and custom configurations. |
Sources: backend/onyx/llm/multi_llm.py:195-210, backend/onyx/chat/README.md:183-188
Configuration and Provider Specifics
Onyx manages various providers through custom_config dictionaries and environment variable injection. Each provider often requires unique handling for parameters like API bases or specialized headers.
Provider Handling Logic
- OpenAI/Reasoning Models: OpenAI models use the
responsesAPI path to ensure reasoning packets are captured correctly. Models like o1/o3 use specificreasoning_effortmappings (Low, Medium, High). - Anthropic (Claude): Supports "thinking" blocks. Onyx avoids sending thinking parameters if the history contains tool-calling turns to avoid cryptographic signature errors.
- Local Providers (Ollama/LM Studio): Requires specific
api_baseadjustments (e.g., ensuring/v1suffix) and explicit API key handling. - AWS Bedrock: Requires stripping tool content from messages when tools are not provided for the current request to avoid "toolConfig" definition errors.
Sources: backend/onyx/llm/multi_llm.py:230-265, backend/onyx/llm/multi_llm.py:440-480
Connection Management
To prevent "Bad file descriptor" errors during concurrent streaming, Onyx creates isolated HTTPHandler instances for true OpenAI models. This ensures that connection pools do not become corrupted when multiple threads (e.g., during deep research tasks) stream simultaneously.
The sequence shows how Onyx manages the global environment lock and isolated HTTP handlers for thread-safe LLM calls.
Sources: backend/onyx/llm/multi_llm.py:535-565, backend/onyx/llm/multi_llm.py:615-630
Cost Tracking and Usage Limits
Onyx implements a cost tracking mechanism for LLM usage, particularly for "Onyx-managed" API keys. After every LLM call (streaming or non-streaming), usage statistics are processed to calculate costs in cents.
- Logic: Costs are calculated based on
prompt_tokensandcompletion_tokensusing provider-specific rates. - Storage: If usage limits are enabled, the calculated cost is incremented in the PostgreSQL database via
UsageType.LLM_COST.
Sources: backend/onyx/llm/multi_llm.py:328-360
Frontend Integration (LLM Selector)
The frontend provides an LLMSelector component, primarily used in the Agent Editor, to allow users to override default models for specific personas.
- Custom LLM Modal: Administrators can configure new providers by providing a Display Name, Provider Name, and Model Name. The system performs a test call (
/api/admin/llm/test) before saving the configuration. - Model Descriptors: Models are identified via a descriptor string (e.g.,
provider/model_name) which is parsed by the UI to set overrides for the backend.
Sources: web/src/refresh-pages/AgentEditorPage.tsx:288-305, web/src/sections/modals/llmConfig/CustomModal.test.tsx:75-100
Error Handling
The integration defines specific exceptions to handle common LLM failures gracefully:
| Exception | Trigger |
|---|---|
LLMTimeoutError | Raised when a LiteLLM Timeout occurs (e.g., socket read timeout between chunks). |
LLMRateLimitError | Raised when the provider returns a 429 status code. |
OnyxError | Global backend error wrapper used to communicate failures (like "Invalid API Key") to the frontend. |
Sources: backend/onyx/llm/multi_llm.py:82-90, AGENTS.md:181-190
Onyx's LiteLLM integration provides a robust, thread-safe, and highly configurable layer that supports a diverse ecosystem of AI models while ensuring consistent performance and cost accountability across the platform.