LlamaCPP Engine
Relevant source files
The following files were used as context for generating this wiki page:
The LlamaCPP Engine is a cross-platform backend for text generation and vision-language tasks within the RunAnywhere SDK ecosystem. It wraps the llama.cpp and ggml libraries to provide high-performance inference on local hardware. The engine supports GGUF models and provides hardware acceleration across various platforms, including Apple Metal, NVIDIA CUDA, and Vulkan-compatible devices.
Sources: engines/llamacpp/CMakeLists.txt:1-5, engines/AGENTS.md, core/README.md
Architecture and Modalities
LlamaCPP functions as a Pattern 1 engine, meaning it bundles its own runtime (ggml) and does not rely on external L1 runtime adapters for its core compute tasks. It implements four specific AI primitives defined in the RunAnywhere plugin ABI.
Supported Primitives
| Primitive | vtable Slot | Description |
|---|---|---|
| GENERATE_TEXT | llm_ops | Large Language Model (LLM) text generation. |
| EMBED | embedding_ops | Generates vector embeddings from text input. |
| VLM | vlm_ops | Vision-Language Model processing using the mtmd extension. |
| RERANK | rerank_ops | Cross-encoder reranking using rank-pooling GGUFs. |
Sources: engines/AGENTS.md, core/README.md, engines/llamacpp/CMakeLists.txt:175-185
Plugin Registration Flow
The engine registers its functionality through a vtable-based plugin system. On platforms supporting dynamic linking, such as Android or Windows, the SDK calls rac_backend_llamacpp_register() to add the engine to the global registry.
The diagram shows the initialization and registration sequence of the LlamaCPP backend into the RunAnywhere core.
Sources: engines/llamacpp/rac_backend_llamacpp_register.cpp:5-15, engines/AGENTS.md
Hardware Acceleration
LlamaCPP provides specialized support for various hardware backends. The build system automatically configures these backends based on the target platform and available SDKs.
Platform Support Matrix
- iOS/macOS: Uses Apple Metal (
GGML_METAL) and Accelerate framework for GPU acceleration. - Android: Primarily CPU-based with NEON optimization for ARM architectures. It supports 16KB page-alignment for Android 15+ compliance.
- Windows: Supports Vulkan as the default GPU path and CUDA for NVIDIA hardware.
- Linux: Uses Vulkan for cross-vendor GPU acceleration (AMD, Intel, NVIDIA).
Sources: engines/llamacpp/CMakeLists.txt:60-170, core/AGENTS.md
Acceleration Configuration
The diagram illustrates how the LlamaCPP engine selects hardware acceleration technologies based on the build target.
Sources: engines/llamacpp/CMakeLists.txt:60-150
Implementation Details
Core Files and Components
The LlamaCPP engine implementation is divided into several specialized source files:
llamacpp_backend.cpp: Core backend logic and integration with the llama.cpp C++ API.rac_llm_llamacpp.cpp: Implementation of the LLM text generation primitive.rac_vlm_llamacpp.cpp: Logic for multimodal vision-language tasks.rac_rerank_llamacpp.cpp: Implementation of the reranking primitive for rank-pooling models.rac_plugin_entry_llamacpp.cpp: The unified ABI plugin entry point that exports the manifest and vtable.
Sources: engines/llamacpp/CMakeLists.txt:174-190, engines/AGENTS.md
Multimodal (VLM) Support
Multimodal support is integrated into the LlamaCPP engine using the mtmd (multimodal) model implementations from the llama.cpp tools directory. This allows the engine to handle image inputs alongside text prompts.
This sequence diagram shows the data flow when the engine processes vision-language tasks.
Sources: engines/llamacpp/CMakeLists.txt:200-220, engines/AGENTS.md
Build Configuration
The LlamaCPP engine is managed via CMake. It uses FetchContent to retrieve the llama.cpp source code from the configured repository and tag defined in the project's VERSIONS file.
| CMake Option | Default | Description |
|---|---|---|
RAC_BACKEND_LLAMACPP | ON | Master toggle to build the LlamaCPP backend. |
GGML_METAL | ON (Apple) | Enables Apple Metal GPU acceleration. |
GGML_VULKAN | Auto | Enables Vulkan GPU acceleration if Vulkan SDK is found. |
GGML_CUDA | OFF | Opt-in toggle for NVIDIA CUDA acceleration (Windows/MSVC). |
GGML_NEON | ON (ARM) | Enables ARM NEON SIMD optimizations. |
Sources: engines/llamacpp/CMakeLists.txt:5-150, core/AGENTS.md
Summary
The LlamaCPP Engine serves as a versatile, local-first inference provider for LLM and VLM tasks within the RunAnywhere SDK. By leveraging the llama.cpp library and providing deep integration with platform-specific GPU technologies like Metal and Vulkan, it ensures efficient AI performance across mobile, desktop, and embedded environments. Its architecture as a self-contained engine plugin allows it to be easily registered or updated without modifying the core SDK infrastructure.