Provider Integration
Relevant source files
The following files were used as context for generating this wiki page:
Provider Integration in M31A is the architectural layer responsible for abstracting communications between the autonomous engine and Large Language Model (LLM) gateways. It provides a unified interface for multiple backends, allowing the system to switch between different models and providers dynamically based on task complexity, cost, or availability.
The system currently supports three primary provider integrations: OpenRouter, Zen, and Nvidia NIM. These integrations handle model discovery, streaming chat completions, and cost estimation while ensuring that API keys are managed securely via the OS keychain rather than plaintext files.
Sources: README.md:38-40, README.md:154-158, internal/core/config/types.go:137-140
Architecture Overview
The integration layer follows a provider pattern where a central interface defines the required behaviors for any LLM gateway. This allows the workflow engine to remain agnostic of the specific API nuances of OpenRouter, Zen, or Nvidia.
The architecture uses a BaseClient to centralize common logic such as health checks and error handling, reducing duplication across specific implementations.
Sources: internal/integrations/provider/interface.go:10-25, CHANGELOG.md:143-144, internal/integrations/provider/base_client.go
The LLMProvider Interface
All integrations must implement the LLMProvider interface. This ensures that the engine can interact with any registered provider using a standard set of methods.
Key Interface Methods
| Method | Purpose |
|---|---|
Name() string | Returns the identifier of the provider (e.g., "openrouter"). |
FetchModels(ctx) | Dynamically discovers available models from the provider API. |
ChatCompletionStream(ctx, req) | Initiates a Server-Sent Events (SSE) stream for model responses. |
EstimateCost(model, usage) | Calculates the USD cost of a specific transaction. |
HealthCheck(ctx) | Validates API connectivity and credential validity. |
Sources: internal/integrations/provider/interface.go:10-25, internal/engine/workflow/integration_test.go:27-51
Provider Management and Fallback
M31A implements an automatic fallback mechanism. If a primary provider degrades or returns errors, the system rotates to the next available provider in the priority list.
The fallback priority and health check behaviors are fully configurable via the config.toml file.
Sources: internal/core/config/types.go:154-162, README.md:154-158, CHANGELOG.md:41-43
Configuration and Credentials
Providers are configured through the ProviderConfig and ProviderCredentialConfig structures. M31A prioritizes security by ensuring API keys are resolved from the OS keychain (Linux Secret Service, macOS Keychain, or Windows Credential Manager).
Configuration Options
| Option | Type | Description |
|---|---|---|
default | string | The primary provider to use on startup. |
auto_fallback | bool | Enables/disables automatic switching on failure. |
fallback_priority | []string | Ordered list (e.g., ["nvidia", "zen", "openrouter"]). |
health_check_timeout_secs | int | Duration before a provider is marked as unreachable. |
openrouter_base_url | string | Optional override for self-hosted or proxied gateways. |
Sources: internal/core/config/types.go:137-166, README.md:200-210, CHANGELOG.md:41-45
Model Arbitrage and Context Handling
The provider integration layer works closely with the Model Arbitrage system to optimize for cost and performance. The engine classifies tasks (trivial, simple, moderate, complex) and requests a model that meets the required context window.
- Context Estimation: Uses
tiktoken-based estimation with EMA (Exponential Moving Average) self-calibration to predict token usage. - AutoDream: When a provider's context window limit is approached (detected via patterns like "input...exceeds"), the system triggers context consolidation to compress the conversation history.
Sources: README.md:154-164, internal/core/config/types.go:174-182, CHANGELOG.md:112-114
Summary
Provider Integration is the bridge that enables M31A's "multi-brain" capability. By abstracting OpenRouter, Zen, and Nvidia behind a unified interface, the system achieves high reliability through automatic fallback and cost efficiency through model arbitrage. This layer ensures that the workflow engine can focus on task execution while the integration layer manages the complexities of LLM API communications, token limits, and secure credential storage.
Sources: README.md:38-42, internal/integrations/provider/interface.go:10-15