Large Language Models (LLM)
Relevant source files
The following files were used as context for generating this wiki page:
Large Language Models (LLM) within this project provide the core text generation capabilities, supporting features such as streaming completions, structured JSON output, and tool calling. The LLM system follows a two-layer architecture consisting of a Service layer for backend dispatch and a Component layer for lifecycle management and event emission.
The LLM functionality is exposed through a stable C ABI (rac_llm_*) and is implemented across multiple platforms including iOS, Android, Windows, macOS, Linux, and WebAssembly. It supports various backends like llama.cpp, QHexRT (Qualcomm Hexagon NPU), and Apple Foundation Models.
Sources: core/README.md, core/AGENTS.md:65-75, bindings/electron/src/api/text.ts:6-12
System Architecture
The LLM architecture separates high-level request handling from low-level inference backends. The Service layer dispatches requests to the appropriate engine vtable based on the model's framework and priority.
Architecture Overview
The diagram shows the flow from platform SDKs through the C ABI to the internal service layer and backend plugins.
Sources: core/README.md, core/AGENTS.md:58-75
Two-Layer Design
- Service Layer (
rac_llm_service.cpp): Manages dispatch logic. It identifies the required backend by looking up models in the registry and resolving the appropriaterac_inference_framework_t. - Component Layer: Owns the model lifecycle and handles operational tasks such as token streaming, cancellation, and analytics event emission.
Sources: core/AGENTS.md:65-75
Key Features
Text Generation
The system supports both synchronous and asynchronous (streaming) text generation. Users can provide a single prompt or an array of chat messages representing a conversation.
Structured Output
The LLM can generate JSON output that satisfies a specific JsonSchema. This ensures that the generated text is constrained to a format that can be reliably parsed by the application.
Sources: idl/llm_service.proto:115-121, bindings/electron/src/api/text.ts:258-264
Tool Calling
The LLM can interact with external functions through tool calling. The LLM Service manages the loop of identifying tool needs, parsing arguments, and incorporating tool results back into the conversation.
This sequence illustrates the tool calling loop managed by the LLM service.
Sources: bindings/electron/src/api/text.ts:13-17, idl/llm_service.proto:168-175
Data Models and API
The LLM Service uses Protobuf for data serialization across the C ABI boundary.
Generation Options
| Option | Type | Description |
|---|---|---|
model_id | string | The unique identifier of the model to use. |
temperature | float | Controls randomness (higher is more random). |
max_tokens | uint32 | The maximum number of tokens to generate. |
stop_sequences | repeated string | Sequences that trigger the end of generation. |
top_p | float | Nucleus sampling threshold. |
Sources: idl/llm_service.proto:58-100
Stream Events
When using streaming API, the service emits events in the LLMStreamEvent format.
| Event Type | Description |
|---|---|
STARTED | Indicates the start of the generation process. |
TOKEN | Contains a delta of generated text. |
THINKING | Contains reasoning or "thought" tokens (if supported). |
TOOL_CALL | Contains tool invocation details. |
COMPLETED | Indicates successful completion with final metrics. |
ERROR | Contains error details if the generation fails. |
Sources: idl/llm_service.proto:141-164, bindings/electron/src/api/text.ts:468-490
Implementation Details
Model Resolution
The service ensures model residency before inference. If the requested model_id is not currently loaded, the service interacts with the Lifecycle Manager to bring the model into memory.
Sources: bindings/electron/src/api/text.ts:316-328, core/AGENTS.md:126-130
Cancellation
Generation can be cancelled via rac_llm_cancel. Internally, this sets an atomic<bool> flag (cancel_requested) that is checked by the inference loop to stop work immediately.
Sources: core/AGENTS.md:113-115, bindings/electron/src/api/text.ts:503-505
Metrics and Usage
The LLMGenerationResult includes detailed performance metrics such as:
input_tokens: Count of tokens in the prompt.output_tokens: Count of generated tokens.tokens_per_second: Throughput of the decoding process.ttft_ms: Time to first token in milliseconds.
Sources: idl/llm_service.proto:123-139
Conclusion
The LLM module serves as the primary text intelligence layer of the project. By utilizing a plugin-based backend architecture and a Protobuf-driven C ABI, it provides a performant, cross-platform interface for advanced AI tasks while maintaining strict control over model lifecycle and resource usage.