Retrieval-Augmented Generation (RAG)
Relevant source files
The following files were used as context for generating this wiki page:
Retrieval-Augmented Generation (RAG) provides a system for grounding Large Language Model (LLM) responses in specific, indexed document data. Within this project, RAG operates as a composed pipeline that orchestrates embedding models for document indexing and retrieval, and LLMs for generating answers based on the retrieved context. The system is designed to run entirely on-device, utilizing C++ commons for the core logic and vector storage while providing high-level SDK interfaces for various platforms including Web, Electron, and mobile.
The RAG architecture supports a split-modality approach. It uses a vector store (typically USearch) for dense retrieval and often includes BM25 for sparse hybrid retrieval. The pipeline supports document ingestion, similarity-based search, and full query-response cycles where the system automatically injects retrieved document chunks into the LLM's prompt.
Sources: AGENTS.md, core/AGENTS.md, idl/rag.proto
System Architecture and Providers
The RAG system is implemented as a two-layer feature. The Component Layer owns the model lifecycle and exposes the public API, while the Service Layer manages the orchestration of backends. Because RAG is a composed pipeline rather than a single primitive, it does not use a single backend vtable but instead coordinates multiple services like embedding_ops and llm_ops.
In the Web SDK, the system can utilize different providers based on environment capabilities:
- NativeRAGSessionProvider: Utilizes the C++ commons native RAG session when WASM exports are available.
- CrossWasmRAGProvider: A fallback that orchestrates an ONNX embedding module and a LlamaCPP generation module across different Emscripten heaps.
- PersistentRAGProvider: Extends CrossWasm with IndexedDB persistence for the vector index.
Sources: core/AGENTS.md, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+RAG.ts:167-240
RAG Data Flow
This diagram illustrates the lifecycle of a RAG query from the user interface through retrieval and generation.
Configuration and Options
The RAG system is configured via the RAGConfiguration message, which defines the models used and the parameters for chunking and retrieval.
RAGConfiguration Fields
| Field | Type | Default | Description |
|---|---|---|---|
embedding_model_id | string | - | The ID of the model used to create vector embeddings. |
llm_model_id | string | - | The ID of the LLM used for answer generation. |
top_k | int32 | 5 | The number of document chunks to retrieve for context. |
score_threshold | float | 0.0 | Minimum similarity score (0.0 to 1.0) to include a chunk. |
chunk_size | int32 | 512 | Number of tokens per document chunk. |
chunk_overlap | int32 | 64 | Token overlap between adjacent chunks. |
rerank_results | bool | false | If true, performs a pointwise rerank of chunks using the LLM. |
Sources: idl/rag.proto:17-48, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+RAG.ts:850-865
Document Ingestion
The ingestion process involves breaking a document into chunks, embedding those chunks into vectors, and storing them in a vector index. Re-ingesting a document with an existing ID replaces its previous chunks.
Query and Search API
The SDK provides two primary ways to interact with the indexed data:
- Search: Returns relevant document chunks (
RAGSearchResult) with similarity scores without generating a natural language answer. - Query: Performs retrieval and then calls the LLM to generate a grounded
RAGResult. This can be performed as a single call or a stream.
RAGResult Structure
The RAGResult contains the final answer and metadata about the retrieval process.
| Field | Type | Description |
|---|---|---|
answer | string | The natural language answer generated by the LLM. |
retrieved_chunks | list | Chunks used to ground the answer. |
context_used | string | The actual string injected into the LLM prompt. |
retrieval_time_ms | int64 | Time taken to embed query and search the index. |
generation_time_ms | int64 | Time taken by the LLM to generate the answer. |
usage | TokenUsage | Token consumption metrics from the LLM. |
Sources: idl/rag.proto:120-141, bindings/electron/src/api/data.ts:133-142
Multi-Query Expansion
The system supports enable_multi_query, which allows the LLM to expand a single user query into several phrasings (up to 8). These queries are searched independently, and results are merged to improve retrieval recall.
Sources: idl/rag.proto:79-88
Internal Implementation Details
Vector Storage (USearch)
The core C++ implementation uses USearch for vector indexing. It dedups embeddings using content-addressing to avoid re-embedding identical content. In the Web environment, if native C++ USearch is unavailable, the system falls back to a TypeScript-managed in-memory index or IndexedDB for persistence.
Sources: core/AGENTS.md, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+RAG.ts:525-545, core/tests/test_rag_e2e.cpp:189-217
Chunker Logic
The DocumentChunker splits text based on separators (like newlines or spaces). It enforces UTF-8 boundaries to ensure characters are not sliced in the middle of multi-byte sequences, particularly for CJK (Chinese, Japanese, Korean) text.
Sources: core/tests/test_rag_e2e.cpp:322-370
Streaming Events
RAG queries can be streamed via RAGStreamEvent. The stream emits TOKEN events for the generated answer and a terminal COMPLETED event containing the full result and retrieved chunks.
Sources: idl/rag.proto:159-178, bindings/web/packages/core/src/Public/Extensions/RunAnywhere+RAG.ts:646-680
The Retrieval-Augmented Generation system provides a robust on-device pipeline for grounding AI responses. By integrating embedding-based retrieval with LLM generation, it allows applications to serve users with accurate, data-backed information without relying on server-side processing.