LLM Providers & Abstraction
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The LLM Providers & Abstraction layer in SecretForge AI serves as a unified interface for interacting with various Large Language Models (LLMs), specifically OpenAI's GPT-4o and local Ollama instances (Llama 3.1). This system abstracts the underlying API complexities, providing a consistent surface for features like natural language command parsing, secret provision intelligence, and project dependency analysis.
By utilizing an abstraction layer, the project ensures high availability through automatic failover mechanisms and enhances developer privacy by preferring local execution via Ollama when available. This architecture supports the project's goal of "AI-powered secret management" by enabling seamless transitions between cloud-based and edge-native AI processing.
Sources: packages/shared/src/ai/llmProvider.ts:1-12, README.md:12-25
Core Architecture & Providers
The system is built around a LLMProvider class that handles configuration, provider selection, and API communication.
Provider Selection Logic
The abstraction layer uses an "auto" selection mode that prioritizes local Ollama instances for privacy and cost-efficiency. If a local Ollama service is detected (via a heartbeat check to /api/tags), it becomes the active provider. If Ollama is unavailable, the system falls back to OpenAI if an API key is configured.
The diagram above illustrates the provider selection and fallback logic.
Sources: packages/shared/src/ai/llmProvider.ts:111-137, SYSTEM_EVALUATION.md:200-210
Supported Models and Configuration
The system defaults to specific high-performance models based on the selected provider.
| Configuration Option | Type | Default Value | Description |
|---|---|---|---|
provider | 'openai' | 'ollama' | 'auto' | 'auto' | Determines which LLM backend to utilize. |
openaiApiKey | string | '' | API key for OpenAI authentication. |
ollamaBaseUrl | string | 'http://localhost:11434' | Endpoint for the local Ollama service. |
model | string | gpt-4o (OpenAI) / llama3.1:70b (Ollama) | The specific model identifier to be used for completions. |
temperature | number | 0.7 | Controls randomness in generation. |
maxTokens | number | 2000 | Limits the length of the LLM response. |
Sources: packages/shared/src/ai/llmProvider.ts:7-25, packages/shared/src/ai/llmProvider.ts:46-55
Natural Language Command Parsing
The NaturalLanguageParser class utilizes the LLM abstraction to translate human-readable queries into structured commands. This allows users to perform secret management tasks through the CLI or web dashboard using natural language.
Parsing Strategy
To optimize performance and reduce API costs, the system employs a two-tier parsing strategy:
- Rule-based Parsing: Fast path using regex and keyword matching for simple intents (e.g., "list", "rotate").
- LLM-based Parsing: Fallback for complex queries, where the system prompts the LLM to return a structured JSON response containing the
intent,confidence, andparameters.
This sequence shows the fallback from rule-based parsing to LLM completion for intent extraction.
Sources: packages/shared/src/ai/llmProvider.ts:311-410, packages/shared/src/ai/llmProvider.ts:427-458
Extracted Intents
| Intent | Description | Parameters Extracted |
|---|---|---|
provision_secret | Creating a new API key | service, environment |
rotate_secret | Rotating an existing key | service, secretId |
list_secrets | Displaying stored secrets | environment |
analyze_project | Scanning for dependencies | N/A |
check_compliance | Validating against standards | framework (e.g., SOC2, GDPR) |
Sources: packages/shared/src/ai/llmProvider.ts:291-309, packages/shared/src/ai/llmProvider.ts:430-440
Integration and Tooling
The LLM abstraction is integrated across multiple project packages to provide intelligent features.
MCP Server Integration
The Model Context Protocol (MCP) server utilizes the AI abstraction to offer tools to external assistants like Claude. These tools include analyze_project, which detects required services from package.json, and provision_api_key, which orchestrates the generation of secure credentials.
Sources: packages/mcp-server/src/index.ts:50-130
Cloudflare Agents
The system leverages Cloudflare Agents for stateful AI interactions. In the API integration tests, endpoints like /api/agent/chat are used to facilitate conversations regarding secret management, enforced by tier-based access control (e.g., Pro tier access).
Sources: packages/api/src/tests/api.integration.test.ts:233-264, README.md:145-160
Web Dashboard Interaction
The Next.js dashboard provides a terminal-like interface where the npx @secretforge/cli init flow is simulated, demonstrating LLM-powered project analysis and service detection in real-time.
Sources: packages/web/app/page.tsx:112-130
Implementation Details
Data Structures
The provider uses standardized interfaces for messages and responses to ensure interoperability between OpenAI and Ollama.
export interface LLMMessage {
role: 'system' | 'user' | 'assistant';
content: string;
}
export interface LLMResponse {
content: string;
usage?: {
promptTokens: number;
completionTokens: number;
totalTokens: number;
};
model: string;
provider: string;
}
Sources: packages/shared/src/ai/llmProvider.ts:16-33
Automatic Fallback Logic
The complete method implements a try-catch block that triggers a provider switch if the primary selection fails while in auto mode.
async complete(messages: LLMMessage[]): Promise<LLMResponse> {
const provider = await this.selectProvider();
try {
if (provider === 'openai') return await this.completeOpenAI(messages);
else return await this.completeOllama(messages);
} catch (error) {
if (this.config.provider === 'auto') {
const fallbackProvider = provider === 'openai' ? 'ollama' : 'openai';
// ... Attempt completion with fallbackProvider
}
throw error;
}
}
Sources: packages/shared/src/ai/llmProvider.ts:60-84
Conclusion
The LLM Providers & Abstraction system provides a robust foundation for SecretForge AI's intelligent features. By abstracting OpenAI and Ollama, the project achieves a balance between cloud-scale power and local-first privacy. The dual-tier natural language parsing and integration with the Model Context Protocol allow for an intuitive developer experience, making secure secret management accessible through simple human commands.