LLM Providers & Routing
Relevant source files
The following files were used as context for generating this wiki page:
LLM Providers & Routing in Cogmo forms the core infrastructure for interacting with Large Language Models. The system is designed to be multi-provider and decoupled, allowing the agent to switch between different AI backends—such as Anthropic, OpenAI, or OpenRouter—without altering the higher-level agentic logic. It employs a layered hub-and-spoke topology where a unified LlmProvider interface abstracts away vendor-specific SDK details.
The routing logic relies on a system-level model_providers table that maps specific model IDs to configured providers. This allows for fine-grained control over model selection, priority, and token limits. By separating the provider infrastructure from user profiles, Cogmo ensures that changes to the backend (e.g., rotating an API key or switching from direct Anthropic to OpenRouter) do not require updates to individual conversation contexts or steering rules. Sources: CLAUDE.md, design/providers.md, design/decisions.md
Architecture Overview
Cogmo uses raw Anthropic and OpenAI SDKs wrapped in canonical provider classes. The architecture avoids heavyweight frameworks to maintain a thin, high-performance agentic loop.
The diagram shows the relationship between the agentic loop, the provider abstraction layer, and the underlying data stores for models and secrets. Sources: design/decisions.md, design/providers.md
Provider Implementations
Cogmo currently supports two primary adapter types to handle various LLM backends:
Anthropic Provider
The AnthropicProvider uses the raw @anthropic-ai/sdk. It supports advanced features such as:
- Prompt Caching: Automatically applies ephemeral cache control to system prompts and tools.
- Thinking Blocks: Maps Anthropic's internal reasoning/thinking blocks to Cogmo's canonical
thinkingcontent type. - Token Counting: Uses the native
countTokensAPI for accurate budget management.
Sources: src/llm/anthropic.test.ts, design/decisions.md
OpenAI-Compatible Provider
This provider covers OpenAI (GPT models), OpenRouter, xAI, and any custom endpoint that follows the /v1/chat/completions protocol. It allows the system to be highly extensible to new models and providers as they emerge. Sources: design/decisions.md, src/setup/wizard.ts:133-145
Data Model & Routing Logic
Routing is governed by two main tables in the database. Unlike other systems, the model strategy is hybrid, allowing the use of both per-token APIs and subscription-based CLI delegation for different task types.
model_providers Table
This table defines how a model string is resolved to a specific infrastructure provider.
| Column | Type | Description |
|---|---|---|
id | UUIDv7 | Primary key |
model | Text | The model ID (e.g., claude-3-7-sonnet) |
provider_id | UUID | Foreign key to llm_providers |
priority | Integer | Selection order if multiple providers offer the same model |
context_window | Integer | Max input tokens (overrides default) |
max_output | Integer | Max output tokens (overrides default) |
| Sources: design/providers.md, design/decisions.md |
Resolver Flow
When the agent requests a model, the system follows a specific resolution hierarchy:
- Database Lookup: Checks
model_providersfor a specific row matching the model and provider. - LiteLLM Snapshot: If DB limits are null, it falls through to a bundled LiteLLM snapshot for conservative defaults.
- System Default: A final fallback is used if no other limits are found.
Sources: design/setup.md, src/setup/wizard.ts:316-324
Setup & Configuration
Cogmo includes a guided setup wizard that handles LLM provider registration. It validates credentials against live APIs before persisting them to the database.
The sequence shows the live validation flow during setup to ensure no invalid credentials reach the database. Sources: src/setup/wizard.ts:150-250, design/setup.md
Technical Specifications
Content Block Mapping
The providers must normalize vendor-specific responses into Cogmo's internal ContentBlock array. This includes translating:
- Text Blocks: Standard text output.
- Tool Use: Requests from the model to execute a specific tool.
- Thinking/Reasoning: Internal model "thought" processes (primarily for Anthropic).
Sources: src/llm/anthropic.test.ts, design/decisions.md
Token Management
- Budgeting: The system tracks
inputTokens,outputTokens,cacheReadTokens, andcacheCreationTokens. - Max Tokens: Defaults to 8192 unless specified otherwise in the routing table or thinking budget.
Sources: src/llm/anthropic.test.ts:160-170, src/llm/anthropic.test.ts:500-515
Conclusion
The LLM Provider & Routing system provides a resilient and vendor-neutral foundation for Cogmo. By abstracting providers through a standardized interface and centralizing routing in the database, the project enables seamless model upgrades and cost optimization without impacting the agent's behavioral logic or conversation history.