Embedded Backend (MLX & CPU)
Relevant source files
The following files were used as context for generating this wiki page:The Embedded Backend is the default, "batteries-included" command generation engine for caro. Its primary purpose is to provide immediate, zero-configuration functionality that works entirely offline. This ensures that caro is a plug-and-play tool right after installation, without any dependency on network connectivity or external services.
To achieve both high performance on specialized hardware and broad cross-platform compatibility, the embedded backend employs a dual-variant architecture. It automatically detects the host platform and selects the optimal inference runtime: the MLX variant for GPU-accelerated performance on Apple Silicon, and the CPU variant using the Candle framework as a universal fallback for other platforms like Linux, Windows, and x86_64 macOS. This backend serves as the foundation of caro's reliability, and also acts as the ultimate fallback for remote backends like Ollama and vLLM.
Sources: specs/004-implement-ollama-and/spec.md:15, 60-63, specs/004-implement-ollama-and/plan.md:18-24
Architecture
The embedded backend is designed as a modular system that abstracts the specific inference runtime (MLX or CPU) behind a common set of traits. The main EmbeddedModelBackend struct acts as a public-facing orchestrator, which internally holds a concrete implementation of the InferenceBackend trait. This design allows the core application to interact with the embedded backend through a consistent interface, regardless of the underlying hardware.
Sources: specs/004-implement-ollama-and/tasks.md:21-34, 131-144, specs/004-implement-ollama-and/plan.md:104-111
Component Overview
The following diagram illustrates the primary components and their relationships within the embedded backend module.
Core Components
EmbeddedModelBackend
This is the main struct that implements the CommandGenerator trait for the rest of the application. It is responsible for:
- Detecting the platform to select the appropriate
ModelVariant(MLX or CPU). - Lazily loading the model on the first
generate_command()call to ensure fast application startup. - Holding the active
InferenceBackend(eitherMlxBackendorCpuBackend). - Dispatching inference requests to the active backend.
- Parsing the raw string output from the model into a structured
GeneratedCommand.
Sources: specs/004-implement-ollama-and/tasks.md:131-159
InferenceBackend Trait
This trait defines the common interface for the platform-specific inference runtimes. It abstracts the details of loading a model and running inference.
| Method | Description |
|---|---|
infer() | Runs the model with a given prompt and returns the generated string. |
load() | Loads the model weights into memory (CPU or GPU). |
unload() | Releases model resources from memory. |
variant() | Returns the ModelVariant (MLX or CPU) of the implementation. |
Sources: specs/004-implement-ollama-and/tasks.md:109-120
ModelVariant Enum
A simple enum to represent the two possible inference runtimes. It includes a detect() method that determines the appropriate variant based on the target OS and architecture at compile time.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum ModelVariant {
MLX, // Apple Silicon GPU
CPU, // Cross-platform Candle
}
Sources: specs/004-implement-ollama-and/tasks.md:102-108
Platform-Specific Implementations
The backend's dual-variant nature is key to its balance of performance and portability.
MlxBackend (Apple Silicon)
This is the high-performance variant optimized for Apple Silicon (macOS on aarch64).
- Technology: Uses the
mlx-rscrate to leverage Apple's Metal framework for GPU acceleration. - Target Platform:
aarch64-apple-darwin. The code is conditionally compiled only for this target. - Performance Goals:
- Initialization: < 100ms.
- Inference: < 2 seconds for a typical command.
- First Token Latency: < 200ms.
- Features: Takes advantage of the Unified Memory Architecture on Apple Silicon for efficient, zero-copy data handling between the CPU and GPU.
Sources: src/backends/embedded/mlx.rs, specs/004-implement-ollama-and/spec.md:102-104, specs/004-implement-ollama-and/tasks.md:160-176
CpuBackend (Cross-Platform)
This is the universal fallback variant that ensures caro runs on any platform supported by the Rust compiler.
- Technology: Uses the
candle-coreandcandle-transformerscrates for CPU-based inference. - Target Platforms: Linux (x86_64, aarch64), Windows (x86_64), and macOS (x86_64).
- Performance Goals: While slower than the MLX variant, it is targeted to complete inference in under 5 seconds.
- Features: Provides a pure Rust, self-contained inference solution with no external system dependencies.
Sources: src/backends/embedded/cpu.rs, specs/004-implement-ollama-and/spec.md:105-106, specs/004-implement-ollama-and/tasks.md:177-190
Data Flow and Logic
The process of generating a command using the embedded backend follows a clear, lazy-loaded sequence to optimize startup performance.
This diagram shows the sequence of operations for a command generation request.
Model and Configuration
Model Management
The embedded backend is designed to work with a specific, pre-selected model to ensure a consistent and reliable offline experience.
- Default Model: Qwen (Qwen2.5-Coder) is the primary model, chosen for its balance of speed, size, and shell command accuracy.
- Model Distribution: Models are not bundled in the binary to keep its size small (<50MB). Instead, a
ModelLoaderutility downloads the required model from the Hugging Face Hub on the first run and stores it in a local cache directory (~/.cache/caro/models/). This allows for offline operation on subsequent runs. - Alternative Models: While Qwen is the default, the CI/CD pipeline also benchmarks against
Phi-3andStarCoder2to continuously evaluate the best option.
Sources: specs/004-implement-ollama-and/spec.md:107, 115, specs/004-implement-ollama-and/tasks.md:40-52, 53-68
Configuration
The behavior of the embedded model can be tweaked via the EmbeddedConfig struct.
| Field | Type | Default | Description |
|---|---|---|---|
temperature | f32 | 0.7 | Controls the randomness of the output. Lower is more deterministic. |
max_tokens | usize | 100 | The maximum number of tokens to generate. |
top_p | f32 | 0.9 | Nucleus sampling parameter. |
stop_tokens | Vec<String> | ["\n\n", "```"] | Sequences that will stop the generation process. |
Sources: specs/004-implement-ollama-and/tasks.md:121-130
Testing
The embedded backend is validated through a series of contract tests that enforce its core requirements. These tests are written before the implementation (TDD RED phase) and must pass for the feature to be considered complete.
Key contract requirements tested in tests/contracts/embedded_backend_contract.rs:
| Contract ID | Requirement | Test Function |
|---|---|---|
CR-EMB-001 | Must operate fully offline with no network calls. | test_offline_operation_no_network_calls |
CR-EMB-002 | Must be available immediately with zero configuration. | test_zero_config_immediate_availability |
CR-EMB-003 | Must meet performance targets for MLX and CPU variants. | test_performance_targets_mlx_vs_cpu |
CR-EMB-004 | Must automatically detect the correct platform variant. | test_platform_detection_automatic |
CR-EMB-006 | Must lazy-load the model on the first inference call. | test_lazy_loading_on_first_inference |
CR-EMB-007 | Must handle model loading failures gracefully. | test_error_handling_model_load_failure |
CR-EMB-009 | Must be thread-safe for concurrent requests. | test_thread_safe_concurrent_requests |
Sources: tests/embedded_backend_contract.rs, specs/004-implement-ollama-and/tasks.md:70-84
Conclusion
The Embedded Backend is a cornerstone of the caro project, delivering on the promise of a fast, safe, and easy-to-use tool that works out-of-the-box. Its dual-variant architecture provides an optimized experience for Apple Silicon users via the MLX backend while ensuring universal access for all other users through the cross-platform CPU backend. By handling model management and abstracting away the complexities of inference, it provides a reliable foundation for command generation and serves as the default engine and ultimate fallback in the caro backend ecosystem.