Adding a New Backend

Relevant source files

The following files were used as context for generating this wiki page:

• CONTRIBUTING.md
• specs/004-implement-ollama-and/tasks.md
• specs/004-implement-ollama-and/plan.md
• specs/002-implement-tdd-green/plan.md
• specs/001-create-a-comprehensive/tasks.md
• exports/session-logs/2025-09-13-readme.txt
• specs/004-implement-ollama-and/spec.md

caro is designed with a modular backend system to support various Large Language Model (LLM) providers for shell command generation. This architecture allows developers to easily add new backends, whether they are local inference engines or remote API-based services. The system is built around a central CommandGenerator trait, ensuring that all backends provide a consistent interface for command generation, availability checks, and metadata retrieval.

The project supports several backend types, including an embedded model for offline use, and optional remote backends like Ollama and vLLM for enhanced performance or quality. This guide outlines the architecture and the Test-Driven Development (TDD) process for contributing a new backend to caro.

Sources: CONTRIBUTING.md, specs/004-implement-ollama-and/spec.md:10-12

Core Architecture

The backend system is designed to be pluggable, with a clear separation of concerns. The core of this system is the CommandGenerator trait, which all backend implementations must satisfy.

Sources: specs/002-implement-tdd-green/plan.md:126-129, exports/session-logs/2025-09-13-readme.txt:168-171

The CommandGenerator Trait

The CommandGenerator trait is the primary contract for all backends. It defines the essential capabilities required for generating commands and integrating with the caro CLI. The async-trait crate is used to allow asynchronous methods within the trait.

Sources: specs/002-implement-tdd-green/plan.md:231-233, specs/004-implement-ollama-and/tasks.md:113-119

The following diagram illustrates the CommandGenerator trait and its relationship with various backend implementations.

The methods of the CommandGenerator trait are detailed below:

Method	Description
generate_command(&self, request: &CommandRequest)	Asynchronously generates a shell command based on the user's request. This is the primary function of any backend.
is_available(&self)	Asynchronously checks if the backend is available and ready to serve requests. For remote backends, this may involve a health check.
backend_info(&self)	Returns a BackendInfo struct containing metadata about the backend, such as its type, model name, and version.
shutdown(&self)	Gracefully shuts down the backend, releasing any resources like network connections or loaded models.

Sources: specs/004-implement-ollama-and/tasks.md:113-125

Backend Types

The system is designed to support multiple types of backends, managed through the BackendType enum.

Backend Type	Description
Embedded	The default, offline-capable backend using a model (like Qwen) compiled into the binary. It has variants for MLX (Apple Silicon) and CPU (Candle).
Ollama	Integrates with a local Ollama service via its HTTP API.
vLLM	Integrates with a remote vLLM OpenAI-compatible API endpoint.
MLX	A specific variant of the embedded backend, optimized for Apple Silicon GPUs.
Mock	A mock implementation used exclusively for testing purposes.

Sources: specs/002-implement-tdd-green/plan.md:155-156, specs/004-implement-ollama-and/plan.md:215-216

Backend Selection and Fallback

caro employs a prioritized selection logic and a robust fallback mechanism to ensure reliability.

This flowchart shows how caro selects which backend to use for a given command.

The selection priority is as follows:

1. CLI Flag: A --backend flag provided in the command line overrides all other settings.
2. Configuration File: If no CLI flag is present, caro checks for a default_backend setting in ~/.config/caro/config.toml.
3. Default: If neither of the above is set, caro defaults to the Embedded model.

A key feature is the fallback system. If a remote backend (Ollama or vLLM) is selected but fails to respond or is unavailable, the system automatically and seamlessly falls back to the Embedded model. This ensures that caro remains functional even with network issues or misconfigured remote services.

Sources: specs/004-implement-ollama-and/spec.md:73-75, 107-109, specs/004-implement-ollama-and/tasks.md:237-259

Development Workflow for a New Backend

Adding a new backend follows a strict Test-Driven Development (TDD) workflow. This ensures that all new backends adhere to the required contracts and are well-tested before integration.

Sources: CONTRIBUTING.md, specs/002-implement-tdd-green/plan.md:88-93

Step 1: Write Failing Contract Tests (RED)

Before writing any implementation code, you must create a new contract test file in the tests/contracts/ directory (e.g., tests/contracts/my_new_backend_contract.rs). These tests should validate all the requirements of the CommandGenerator trait for your new backend.

These tests must fail initially, as the implementation does not yet exist. This is the "RED" phase of TDD.

Example of a failing test for a new VllmBackend:

// tests/contract/vllm_contract.rs
#[tokio::test]
async fn test_vllm_backend_implements_command_generator() {
    let backend = VllmBackend::new(config); // This will fail to compile initially
    let request = CommandRequest::new("list files");
    let result = backend.generate_command(&request).await;
    assert!(result.is_ok());
}

Sources: CONTRIBUTING.md, specs/001-create-a-comprehensive/tasks.md:31-33

Step 2: Create the Backend Module

Create a new module file for your backend within the src/backends/ directory (e.g., src/backends/vllm.rs). Add the necessary struct definition for your backend.

// src/backends/vllm.rs
pub struct VllmBackend {
    // fields for connection, config, etc.
}

Sources: specs/001-create-a-comprehensive/tasks.md:81-82, specs/004-implement-ollama-and/tasks.md:18-22

Step 3: Implement the CommandGenerator Trait (GREEN)

Implement the CommandGenerator trait for your new backend struct. Write the minimal amount of code required to make your failing contract tests pass. This is the "GREEN" phase.

// src/backends/vllm.rs
use async_trait::async_trait;
// ... other imports

#[async_trait]
impl CommandGenerator for VllmBackend {
    async fn generate_command(&self, request: &CommandRequest) -> Result<GeneratedCommand, GeneratorError> {
        // Implementation logic here
    }

    // ... implement other trait methods
}

Sources: specs/002-implement-tdd-green/plan.md:231-233, CONTRIBUTING.md

Step 4: Integrate into Backend Selector

Once your backend is implemented and its contract tests are passing, integrate it into the backend selection logic. This typically involves updating the BackendSelector in src/backends/selector.rs to recognize and instantiate your new backend based on the BackendType enum and configuration.

Sources: specs/004-implement-ollama-and/tasks.md:260-272

Step 5: Add Integration and End-to-End Tests

Finally, add integration tests in the tests/integration/ directory to verify that your backend works correctly within the full application flow. These tests should cover scenarios from the quickstart.md document, including successful generation, error handling, and interaction with other modules like configuration and safety validation.

Sources: specs/001-create-a-comprehensive/tasks.md:37-45

Conclusion

The modular backend architecture is a cornerstone of caro, enabling flexibility and extensibility. By following the CommandGenerator trait contract and the TDD workflow, contributors can easily expand caro's capabilities to support new and emerging LLM services. This design ensures that the core application remains robust and reliable while fostering a community-driven ecosystem of backend integrations.

Sources: CONTRIBUTING.md