Embeddings Generation & Service
Relevant source files
The following files were used as context for generating this wiki page:
The Embeddings Generation & Service module is responsible for transforming raw text data into high-dimensional vector representations, enabling semantic search and retrieval within the RAG (Retrieval-Augmented Generation) pipeline. This service interfaces with external providers like Ollama or dedicated Hugging Face model servers to process document chunks and user queries.
This system ensures that technical documentation is indexed efficiently using a character-based chunking strategy and supports "instruction-aware" embeddings to optimize retrieval performance for modern models like Qwen3. Sources: Plan.md:4-10, backend/app/config.py:33-40
Service Architecture & Integration
The project utilizes a decoupled architecture where the backend application orchestrates data flow while the heavy computation of vector generation is offloaded to a dedicated embedding server.
The diagram shows the flow of text data from the frontend settings to the document processing pipeline, which interacts with external embedding servers to populate the vector database. Sources: backend/app/config.py:35-45, Plan.md:145-155
Core Components
- Embedding Service: Manages the lifecycle of an embedding request, including batching, prefix application, and error handling. Sources: backend/app/config.py:76-80
- External Provider: Typically Ollama or a TEI (Text Embeddings Inference) compatible endpoint. Sources: CONTRIBUTING.md:28, backend/app/config.py:36
- LanceDB Integration: Vectors are stored with a
cosinedistance metric by default to allow for high-performance similarity searches. Sources: backend/app/config.py:64-67
Processing Logic & Instruction Prefixes
To support "instruction-aware" models (e.g., Qwen3-Embedding-4B), the service applies specific prefixes to text before generation. This distinguishes between document passages meant for storage and queries meant for retrieval. Sources: Plan.md:103-108
Prefix Implementation
The service prepends a specific instruction string to the text content based on the context:
| Context | Prefix Setting | Default Value (Qwen Example) |
|---|---|---|
| Indexing | embedding_doc_prefix | Instruct: Represent this technical documentation passage for retrieval.\nDocument: |
| Querying | embedding_query_prefix | Instruct: Retrieve relevant technical documentation passages.\nQuery: |
Sources: backend/app/config.py:70-74, Plan.md:113-118
Character-Based Chunking
The system has transitioned from token-based to character-based units to align UI labels with backend processing. This prevents mismatch in expected chunk sizes. Sources: Plan.md:33-40
- Chunk Size:
chunk_size_chars(Default: 1200 chars). - Chunk Overlap:
chunk_overlap_chars(Default: 120 chars).
Sources: backend/app/config.py:57-61
Performance & Adaptive Batching
To optimize network throughput and handle model server limits, the service utilizes adaptive batching.
- Batch Size: Defaults to 64 texts per request, capped at 128 for compatibility with TEI. Sources: backend/app/config.py:76-78
- Concurrency: Supports concurrent batch processing (defaulting to 4 concurrent batches) to maximize GPU utilization on the embedding server. Sources: backend/app/config.py:89-93
- Adaptive Fallback: If a batch fails due to token overflow or server error, the system performs a sub-batch retry. Sources: backend/app/config.py:80-82
Batching Sequence
The sequence illustrates how the service batches multiple text segments and applies instruction prefixes before communicating with the external model server. Sources: backend/app/config.py:76-82, Plan.md:113-118
Validation & Integrity
The system performs strict validation of embedding dimensions at startup to prevent "garbage retrieval" caused by model mismatches. Sources: Plan.md:143-148
Dimension Validation
- Probe: On startup, a 1-item embedding request is sent for a probe string.
- Detection: The system detects the length of the resulting vector (Default: 1024).
- Comparison: The detected dimension is compared against the
EMBEDDING_DIMsetting and the existing LanceDB table schema. - Enforcement: If a mismatch is detected, ingestion and search are blocked with a clear "reindex required" error.
Sources: backend/app/config.py:48-52, Plan.md:150-158
Settings Summary
The following table summarizes the primary configuration options for the embedding system:
| Key | Type | Default Value | Description |
|---|---|---|---|
ollama_embedding_url | String | http://.../v1/embeddings | The endpoint for vector generation. |
embedding_model | String | microsoft/harrier-oss-v1-0.6b | Model identifier for the service. |
vector_metric | Enum | cosine | Metric used for distance calculation. |
embedding_batch_size | Integer | 64 | Max texts per API request. |
embedding_dim | Integer | 1024 | Expected vector dimensionality. |
Sources: backend/app/config.py:35-52, backend/app/config.py:64-78
Conclusion
The Embeddings Generation & Service is a critical bridge between unstructured data and semantic search. By implementing character-based chunking, instruction-aware prefixes, and strict dimension validation, the system maintains a high standard of retrieval accuracy and performance, ensuring that the RAG pipeline remains robust across different embedding models and scales. Sources: Plan.md:165-175, specs/kms-content-search-rag/SPEC.md:15-20