LLM Integration
Relevant source files
The following files were used as context for generating this wiki page:
The LLM Integration system in KnowledgeVault provides a dual-model architecture designed to balance high-fidelity reasoning with low-latency responsiveness. The system orchestrates interactions between a "Thinking" model (typically larger models like Qwen 2.5 32B/72B) for complex RAG synthesis and an "Instant" model (typically smaller models like Nemotron-3 4B) for rapid query transformation and title generation.
This integration serves as the core reasoning engine for the RAG pipeline, handling everything from hypothetical document generation (HyDE) to automated wiki curation and chat stream processing.
Sources: README.md:32-48, backend/app/config.py:34-46
Dual-Mode Architecture
The project implements a tiered LLM strategy to optimize for different workloads. This is managed via two primary service endpoints and their respective models.
Chat Modes
- Thinking Mode: Used for primary RAG response generation. It utilizes a comprehensive prompt builder that incorporates document chunks, memories, and wiki claims.
- Instant Mode: A latency-optimized path that skips expensive auxiliary LLM calls (like retrieval evaluation and step-back transformation) to provide faster responses. It is primarily used for title generation and lightweight chat turns.
Sources: backend/app/config.py:34-46, backend/app/config.py:165-179, specs/security-ux-hardening/SPEC.md:46-51
Service Configuration
| Configuration Option | Default Value | Description |
|---|---|---|
ollama_chat_url | http://host.docker.internal:11434 | Endpoint for the primary Thinking model. |
chat_model | gemma-4-26b-a4b-it-apex | The default model used for complex RAG synthesis. |
instant_chat_url | http://host.docker.internal:1234 | Endpoint for the low-latency Instant model. |
instant_chat_model | nvidia/nemotron-3-nano-4b | The model used for rapid transformations and auto-titling. |
default_chat_mode | thinking | Determines which mode is used for new chat sessions. |
Sources: backend/app/config.py:27-41, README.md:143-152
Security and Prompt Hardening
To mitigate prompt injection risks (CWE-1333), the system implements structural isolation for all untrusted content. This content, including document chunks, user memories, and queries, is wrapped in specific XML tags before being injected into the LLM system prompt.
XML Content Boundaries
The system prompt includes a "SECURITY BOUNDARY" directive that instructs the LLM to treat content within these tags as data-only and never to follow instructions embedded within them.
| Tag | Purpose | Content Source |
|---|---|---|
<document> | Retrieved evidence | Vector store document chunks |
<memory> | User context | SQLite memory records |
<user_query> | Transformed input | User search queries for HyDE/Step-back |
<user_message> | Title generation | First message of a session |
Sources: specs/security-ux-hardening/SPEC.md:27-45
Flow of Prompt Construction
The following diagram illustrates how the PromptBuilderService aggregates various data sources into a hardened prompt.
A flowchart showing the multi-stage prompt construction process with XML tagging for security boundaries.
Sources: specs/security-ux-hardening/SPEC.md:27-45, backend/app/services/wiki_curator.py:112-140
LLM Wiki Curator
The Wiki Curator is an optional LLM-assisted module that runs after deterministic extraction. It uses a small instruct model to propose new wiki claims, entities, and relationships that might have been missed by standard processing.
Curator Logic and Verification
The curator operates under a "Trust Boundary" where candidates are only accepted as active wiki claims if their source_quote is verifiable within the specific chunk cited.
- Candidate Generation: The model reads source chunks and proposes structured JSON output.
- Verification: The system performs a normalized substring check or a fuzzy match (threshold 92%) to ensure the quote exists in the source.
- Status Assignment: Claims land as
needs_reviewby default (Draft mode) oractiveif the quote is verified (Active-if-verified mode).
Sources: backend/app/services/wiki_curator.py:12-32, backend/app/services/wiki_curator.py:237-260
Curator Execution Parameters
| Parameter | Configuration | Default / Range |
|---|---|---|
| Concurrency | wiki_llm_curator_concurrency | 1 - 4 parallel calls |
| Max Input | wiki_llm_curator_max_input_chars | 1000 - 24000 chars |
| Mode | wiki_llm_curator_mode | draft, active_if_verified |
| Quote Match | wiki_llm_curator_require_quote_match | Boolean (Default: True) |
Sources: backend/app/config.py:348-369, backend/app/services/wiki_curator.py:146-168
Streaming and SSE Integration
The LLM response is delivered to the frontend via a Server-Sent Events (SSE) stream. This stream includes not only the generated text but also metadata regarding the evidence used during the generation.
SSE Stream Components
- Content Blocks: Streamed answer text chunks.
- Done Event: Contains final metadata arrays including
sources,memories_used,wiki_used, andkms_used. - Citation Repair: If the backend detects invalid citations, it sends a
repaired_contentfield in the final event to reconcile the message before persistence.
A sequence diagram detailing the streaming interaction between the frontend, backend, and LLM service, including the final metadata delivery.
Sources: frontend/src/lib/api.ts:767-840, specs/wire-wiki-chat/SPEC.md:121-135
Conclusion
The LLM Integration provides a robust framework for RAG operations by strictly separating trusted system instructions from untrusted document data through XML tagging. By utilizing a dual-model approach, KnowledgeVault maintains high synthesis quality for complex queries while ensuring the user interface remains responsive through the Instant mode and SSE streaming.