Remote Backend: vLLM
Relevant source files
The following files were used as context for generating this wiki page:
The vLLM backend is an optional, remote command generation engine for caro. It allows users to leverage high-performance GPU clusters running vLLM for generating shell commands, serving as an alternative to the default embedded model. This backend communicates with any vLLM server that exposes an OpenAI-compatible API endpoint.
The primary goal of this backend is to provide access to larger, potentially more powerful models for enhanced generation quality, while integrating seamlessly into caro's existing safety and execution pipeline. It is designed with fallback mechanisms to ensure caro remains functional even if the remote service is unavailable.
Sources: specs/004-implement-ollama-and/spec.md:21-22, 60, exports/session-logs/2025-09-13-readme.txt:101-105
Architecture and Components
The vLLM backend is a modular component that implements the CommandGenerator trait. This ensures it provides a consistent interface for command generation, availability checks, and shutdown procedures, just like other backends (e.g., Embedded, Ollama).
Sources: specs/004-implement-ollama-and/plan.md:310-313, specs/004-implement-ollama-and/data-model.md:155-157
Core Components
| Component | Description | File Path |
|---|---|---|
VllmBackend | The main struct that manages the connection, authentication, and API communication with the vLLM server. It implements the CommandGenerator trait. | src/backends/vllm.rs |
BackendConnection | A shared struct that handles the HTTP connection state, health tracking, and latency monitoring for the remote endpoint. | src/backends/connection.rs |
RetryPolicy | A configurable struct that defines the exponential backoff strategy for handling transient network failures. | src/backends/retry.rs |
VllmRequest | A serializable struct representing the JSON payload sent to the vLLM server's completion endpoint. | src/backends/vllm.rs |
VllmResponse | A deserializable struct representing the JSON response received from the vLLM server. | src/backends/vllm.rs |
Sources: specs/004-implement-ollama-and/data-model.md:155-181, 91-110, 112-129
Class Diagram
The diagram below shows how VllmBackend fits into the CommandGenerator trait system.
Configuration
The vLLM backend can be configured via CLI flags, environment variables, or a configuration file. The CLI flag has the highest priority.
Sources: specs/004-implement-ollama-and/spec.md:31-32, exports/session-logs/2025-09-13-readme.txt:60-62, 131-135
CLI Options
| Option | Description | Default |
|---|---|---|
--backend vllm | Selects the vLLM inference backend. | mlx |
--endpoint <URL> | Sets the remote vLLM server endpoint. | http://localhost:8000 |
Sources: exports/session-logs/2025-09-13-readme.txt:60-62
Configuration File (~/.caro/config.toml)
The vLLM backend can be configured in the user's configuration file for persistent settings.
[backend.vllm]
base_url = "https://api.example.com"
model_name = "codellama/CodeLlama-7b-hf"
api_key = "sk-..." # Optional Bearer token
enabled = true
Sources: specs/004-implement-ollama-and/data-model.md:267-276, specs/004-implement-ollama-and/spec.md:33
Environment Variables
| Variable | Description | Default |
|---|---|---|
CMDAI_BACKEND | Default backend to use. Set to vllm. | mlx |
CMDAI_ENDPOINT | Default endpoint for remote backends. | http://localhost:8000 |
Sources: exports/session-logs/2025-09-13-readme.txt:131-135
API Interaction
The backend communicates with a vLLM server using its OpenAI-compatible API, specifically the /v1/completions endpoint.
Sources: specs/004-implement-ollama-and/plan.md:214-216
Command Generation Flow
The following sequence diagram illustrates the process of generating a command using the vLLM backend.
Request Payload (VllmRequest)
This struct is serialized to JSON and sent as the body of the POST request to the vLLM server.
| Field | Type | Description |
|---|---|---|
model | String | The model identifier to use for generation (e.g., "meta-llama/Llama-2-7b-hf"). |
prompt | String | The user's natural language prompt. |
max_tokens | u32 | The maximum number of tokens to generate. Default is 100. |
temperature | Option<f32> | Sampling temperature. Default is 0.7. |
top_p | Option<f32> | Nucleus sampling parameter. Default is 0.95. |
n | u32 | Number of completions to generate. Always 1 for caro. |
stream | bool | Whether to stream the response. Always false for caro. |
stop | Vec<String> | A list of sequences to stop generation at. |
Sources: specs/004-implement-ollama-and/data-model.md:223-236
Response Payload (VllmResponse)
caro expects a JSON response from the vLLM server, which is deserialized into this structure.
| Field | Type | Description |
|---|---|---|
id | String | A unique identifier for the completion. |
object | String | The type of object, typically "text_completion". |
created | u64 | Unix timestamp of when the completion was created. |
model | String | The model that was used for the completion. |
choices | Vec<VllmChoice> | A list of completion choices. caro uses the first one. |
usage | VllmUsage | Token usage statistics for the request. |
The VllmChoice struct contains the actual generated command in the text field.
Sources: specs/004-implement-ollama-and/data-model.md:238-255
Error Handling and Fallback
The vLLM backend is designed to be resilient. If the remote vLLM server is unavailable or fails, caro will automatically fall back to the embedded model to ensure the user can still generate commands.
Sources: specs/004-implement-ollama-and/contracts/vllm-backend.md, specs/004-implement-ollama-and/spec.md:46
Fallback Flow
Key behaviors:
- Connection Failure: If the vLLM endpoint is unreachable,
carowill respect theRetryPolicyand then fall back to the embedded model. - Authentication Failure: On receiving a 401 or 403 status code, the backend will fall back immediately without retrying.
- Insecure Connection Warning: If the configured
base_urluseshttp://, a warning is logged suggesting the use ofhttps://for production environments.
Sources: specs/004-implement-ollama-and/contracts/vllm-backend.md
Contract and Testing
The behavior of the vLLM backend is formally defined by an API contract and verified through a suite of contract tests.
API Contract
The contract, defined in specs/004-implement-ollama-and/contracts/vllm-backend.md, specifies several key requirements:
- FR-NEW-001: Must fall back to the embedded model on connection failure.
- FR-NEW-002: Must fall back without retry on authentication failure.
- FR-NEW-003: Must handle API key authentication via a Bearer token.
- FR-NEW-004: Must use the configured
RetryPolicyfor transient errors. - FR-NEW-005: Must warn users when connecting to an insecure (HTTP) endpoint.
Sources: specs/004-implement-ollama-and/contracts/vllm-backend.md
Contract Tests
The file tests/vllm_backend_contract.rs contains tests that enforce the API contract. These tests use mock servers to simulate various scenarios, including successful generation, connection failures, authentication errors, and timeouts, ensuring the fallback and retry logic works as specified.
Sources: tests/vllm_backend_contract.rs, specs/004-implement-ollama-and/plan.md:175-177