Remote Backend: Ollama
Relevant source files
The following files were used as context for generating this wiki page:
The Ollama backend provides caro with the ability to connect to a local Ollama HTTP API service for command generation. It is designed as an optional enhancement, allowing users to leverage potentially more powerful models running locally via Ollama. This backend is part of a multi-backend system that prioritizes user-configured remote services but includes a robust fallback mechanism to the default embedded model if the remote connection fails.
This integration ensures that caro can offer higher-quality command generation when an Ollama service is available, without sacrificing its core "batteries-included" principle of working offline out-of-the-box. The system is designed to be resilient, automatically handling connection failures and retries before seamlessly falling back to the embedded model.
Sources: specs/004-implement-ollama-and/spec.md:10, 100-103, specs/004-implement-ollama-and/plan.md:19-22
Architecture and Role
The Ollama backend is one of several optional remote backends, alongside vLLM. Its primary role is to provide an interface for local command generation using an external Ollama server. The system follows a strict priority order for backend selection, ensuring a predictable and configurable user experience.
Sources: specs/004-implement-ollama-and/spec.md:100-103, specs/004-implement-ollama-and/plan.md:23, 212-215
Backend Selection and Fallback Flow
The following diagram illustrates the logic caro uses to select a command generation backend, highlighting the position of the Ollama backend and the fallback mechanism.
This diagram shows the three-tier priority system for backend selection: CLI flag, configuration file, and automatic selection, with a final fallback to the embedded model on remote backend failure.
Sources: specs/004-implement-ollama-and/spec.md:100-103, 110-112, specs/004-implement-ollama-and/plan.md:212-215
Configuration
Users can configure the Ollama backend via the ~/.caro/config.toml file or by using CLI flags. The CLI flag always takes the highest priority.
Sources: specs/004-implement-ollama-and/spec.md:80-83, exports/session-logs/2025-09-13-readme.txt:121-125
TOML Configuration
The following settings are available in ~/.caro/config.toml for the Ollama backend.
| Parameter | Type | Description | Default Value |
|---|---|---|---|
preferred_backend | String | Sets the default backend. Set to "ollama" to prioritize it. | "auto" |
ollama_url | String | The base URL for the Ollama API service. | "http://localhost:11434" |
ollama_model | String | The model name to use from the Ollama service. | "codellama:7b" |
Sources: specs/004-implement-ollama-and/data-model.md:321-337, specs/004-implement-ollama-and/spec.md:82
CLI Usage
The Ollama backend can be selected or configured directly from the command line.
# Use Ollama backend for a single command
caro --backend ollama "your prompt"
# Use Ollama with a specific endpoint
caro --backend ollama --endpoint http://localhost:11434 "your prompt"
Sources: exports/session-logs/2025-09-13-readme.txt:121-125
Core Implementation
The Ollama backend is encapsulated within the OllamaBackend struct, which implements the CommandGenerator trait. This struct manages the connection, request/response cycle, and error handling.
Sources: specs/004-implement-ollama-and/data-model.md:200-205
Key Components
The diagram below shows the main structs involved in the Ollama backend implementation and their relationships.
This class diagram illustrates that OllamaBackend implements the CommandGenerator trait and is composed of a BackendConnection for health tracking and an RetryPolicy for resilient communication.
Sources: specs/004-implement-ollama-and/data-model.md:129-148, 158-171, 200-227, 401-409
The OllamaBackend struct contains the following fields:
| Field | Type | Description |
|---|---|---|
connection | BackendConnection | Manages the HTTP connection state and health tracking. |
model | String | The name of the Ollama model to use (e.g., "codellama:7b"). |
retry_policy | RetryPolicy | Configuration for exponential backoff on request failures. |
timeout | Duration | The timeout for a single generation request (default: 30s). |
temperature | f32 | The sampling temperature for generation (default: 0.7). |
max_tokens | Option<u32> | The maximum number of tokens to generate (default: 100). |
Sources: specs/004-implement-ollama-and/data-model.md:206-213
API Interaction
The OllamaBackend communicates with the Ollama server via its /api/generate endpoint. It sends a structured request and parses the JSON response.
Sources: specs/004-implement-ollama-and/plan.md:127-130, specs/004-implement-ollama-and/data-model.md:343-389
Request/Response Flow
This sequence diagram shows the interaction between caro and an Ollama server for a single command generation request.
Data Structures
OllamaRequest
This structure is serialized to JSON and sent as the body of the POST request to /api/generate.
#[derive(Debug, Clone, Serialize)]
pub struct OllamaRequest {
pub model: String,
pub prompt: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub system: Option<String>,
pub stream: bool, // Always false for caro
#[serde(skip_serializing_if = "Option::is_none")]
pub options: Option<OllamaOptions>,
}
#[derive(Debug, Clone, Serialize)]
pub struct OllamaOptions {
#[serde(skip_serializing_if = "Option::is_none")]
pub temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub num_predict: Option<u32>,
}
Sources: specs/004-implement-ollama-and/data-model.md:345-365
OllamaResponse
This structure is deserialized from the JSON response from the Ollama server.
#[derive(Debug, Clone, Deserialize)]
pub struct OllamaResponse {
pub model: String,
pub created_at: String,
pub response: String, // Contains the generated command
pub done: bool,
#[serde(default)]
pub context: Vec<i32>,
#[serde(default)]
pub total_duration: u64, // Nanoseconds
}
Sources: specs/004-implement-ollama-and/data-model.md:371-384
Resilience and Error Handling
The Ollama backend is designed for resilience, with built-in health checking, retry logic, and a fallback mechanism.
Sources: specs/004-implement-ollama-and/data-model.md:129-171
Health Status
The BackendConnection struct maintains a HealthStatus for the remote service, which transitions based on request outcomes.
This state machine shows how the health status of the Ollama connection changes based on success and failure counts, allowing caro to make intelligent decisions about using the backend.
Sources: specs/004-implement-ollama-and/data-model.md:393-409
Retry Policy
If a request fails due to a transient network error, the configured RetryPolicy is enacted. The default policy attempts up to 3 retries with exponential backoff and jitter to avoid overwhelming the server.
Sources: specs/004-implement-ollama-and/data-model.md:158-171
Summary
The Ollama backend serves as a powerful, optional enhancement for caro, enabling users to leverage local Ollama instances for high-quality command generation. It is seamlessly integrated into caro's multi-backend architecture, featuring robust configuration options, resilient error handling with retries, and an automatic fallback to the embedded model. This ensures caro remains both powerful for advanced users and reliable for everyone, even in offline or misconfigured environments.
Sources: specs/004-implement-ollama-and/spec.md:10, 110-112, specs/004-implement-ollama-and/plan.md:19-23