Vision Language Models (VLM) & OCR
Relevant source files
The following files were used as context for generating this wiki page:
Vision Language Models (VLM) and Optical Character Recognition (OCR) provide on-device image understanding and text extraction capabilities. The VLM module enables multi-modal inference, allowing models to process visual data alongside text prompts to generate descriptions, answer questions about images, or extract structured information. The OCR module focuses specifically on identifying and extracting text content from visual inputs.
These features run natively through the C++ core (runanywhere-commons) using backends like llamacpp and qhexrt. High-level SDKs provide ergonomic wrappers around these core services, supporting multiple image formats including raw RGB buffers, encoded bytes, and file paths.
Sources: core/src/features/vlm/rac_vlm_service.cpp, bindings/react-native/example/src/services/VLMService.ts, bindings/electron/test/feature/vlm.feature.test.ts:7-13
Architecture and Data Flow
The Vision system follows a layered architecture where platform-specific SDKs send Protobuf-encoded requests to the C++ core. The rac_vlm_service and rac_ocr_service act as dispatchers, routing requests to the appropriate engine vtable (e.g., llamacpp for GGUF models).
VLM Request Flow
When you initiate a VLM request, the system resolves the required model through the registry and prepares the visual input for the backend.
The diagram shows the transition from high-level SDK calls through Protobuf encoding to native engine execution.
Sources: core/src/features/vlm/rac_vlm_service.cpp:1-40, bindings/react-native/example/src/services/VLMService.ts:30-50
Image Input Handling
The SDK supports three primary ways to provide visual data to the models. All inputs are converted into a VLMImage Protobuf message before reaching the native core.
| Input Type | Description | Source Field |
|---|---|---|
| File Path | A local path to an image file (e.g., PNG, JPEG). | image_path |
| Encoded Bytes | Raw compressed image data (e.g., a Buffer or ByteString). | encoded_bytes |
| Raw RGB | Uncompressed RGB pixel data; requires width and height. | raw_rgb |
Sources: idl/vlm_options.proto:35-45, bindings/electron/test/feature/vlm.feature.test.ts:89-110
Vision Language Models (VLM)
The VLM service processes multi-modal inputs to generate text. It requires a vision model (like smolvlm-256m) and often a vision projector (e.g., mmproj.gguf) resident in the native lifecycle store.
Configuration Options
VLM inference is configured via the VLMOptions message, which controls the generation behavior and visual processing parameters.
| Option | Type | Description |
|---|---|---|
max_tokens | uint32 | Maximum number of tokens to generate. |
temperature | float | Controls randomness (0.0 is deterministic). |
top_p | float | Nucleus sampling threshold. |
stop_sequences | string[] | Tokens that trigger the end of generation. |
image_position | enum | Position of image relative to text (Before/After/Inside). |
Sources: idl/vlm_options.proto:10-30, bindings/electron/test/feature/vlm.feature.test.ts:40-50
VLM Streaming
VLM results are delivered as a stream of events. The VLMStreamingResult provides real-time updates as the model processes the image and generates text.
The sequence shows the asynchronous nature of VLM inference and the event-driven communication between the native core and the application.
Sources: core/src/features/vlm/rac_vlm_service.cpp:50-80, bindings/react-native/example/src/services/VLMService.ts:68-80
Optical Character Recognition (OCR)
The OCR service extracts text from images. It uses a structured OCRRequest and returns an OCRResponse containing recognized text blocks and their locations.
OCR Components
The OCR process involves model resolution and specialized vtable dispatch. Unlike general VLM, OCR is optimized for high-fidelity text extraction.
- rac_ocr_service_t: The native internal structure managing OCR state and engine routing.
- OCRRequest: Protobuf message containing the image and target language/options.
- OCRResponse: Protobuf message containing extracted text, confidence scores, and bounding boxes.
Sources: core/src/features/ocr/rac_ocr_service.cpp:10-35, idl/ocr.proto:5-25
// core/src/features/ocr/rac_ocr_service.cpp:20-25
// Example of OCR internal dispatch logic
rac_status_t rac_ocr_process(rac_ocr_service_t* service, const rac_ocr_request_t* request, rac_ocr_response_t* response) {
if (!service || !service->vt->ocr_ops) return RAC_ERROR_NOT_SUPPORTED;
return service->vt->ocr_ops->process(service->impl, request, response);
}
Sources: core/src/features/ocr/rac_ocr_service.cpp:20-28
Implementation Details
Model Loading for Vision
Vision models often require multiple artifacts. For example, smolvlm-256m requires both a primary model.gguf and an mmproj.gguf projector. The C++ rac_vlm_service ensures both are correctly loaded before inference starts. If a vision model is loaded without its projector, the service returns an error when processing images.
Sources: bindings/electron/test/feature/vlm.feature.test.ts:40-60, core/src/features/vlm/rac_vlm_service.cpp:45
Hardware Acceleration
The VLM system leverages platform-specific hardware:
- WebGPU: Used in browser environments for accelerated VLM kernels.
- QHexRT: Utilizes the Qualcomm Hexagon NPU on Snapdragon devices for high-efficiency mobile vision.
- Metal/CUDA: Used via
llamacppfor desktop and iOS GPU acceleration.
Sources: bindings/react-native/example/src/services/VLMService.ts:15-25, bindings/electron/test/feature/vlm.feature.test.ts:15-20
Error Handling
The vision services return structured rac_status_t codes. Common errors include:
RAC_ERROR_NOT_SUPPORTED: The current engine does not support VLM/OCR primitives.RAC_ERROR_MODEL_NOT_FOUND: The required vision model or projector is not in the registry.RAC_ERROR_INVALID_ARGUMENT: Occurs if raw RGB data is provided without valid width/height dimensions.
Sources: core/src/features/vlm/rac_vlm_service.cpp:55-65, bindings/electron/test/feature/vlm.feature.test.ts:105-115
Summary
The VLM and OCR modules provide a unified interface for on-device computer vision tasks. By routing requests through a C++ core to specialized engines like llamacpp and qhexrt, the system maintains high performance across mobile, desktop, and web platforms. The use of Protobuf for all multi-modal requests ensures type-safety and consistency across the different SDKs in the RunAnywhere ecosystem.