Vector Search & Embeddings
Relevant source files
The following files were used as context for generating this wiki page:
Vector Search and Embeddings represent the core semantic intelligence layer of the SecondLayer platform. This system enables the application to move beyond simple keyword matching to understand the legal context and meaning of user queries, court decisions, and legislative texts. By converting text into high-dimensional numerical vectors (embeddings), the system can perform similarity searches that find relevant legal precedents and procedural norms even when the specific terminology varies.
The infrastructure relies on OpenAI's embedding models for vector generation and Qdrant as the high-performance vector database for storage and retrieval. This system is integrated across multiple services, including legal document analysis, the dynamic template system, and semantic legislation search.
Sources: CLAUDE.md, mcp_backend/src/services/template-system/README.md
Architecture and Components
The Vector Search architecture follows a pipeline where raw legal text is processed, embedded, and indexed for future retrieval.
This diagram shows the data flow from raw legal text through the embedding process into the Qdrant database, and the subsequent retrieval flow for user queries.
Sources: CLAUDE.md, mcp_backend/src/services/legislation-service.ts
Core Services
- EmbeddingService: The primary interface for interacting with LLM providers (specifically OpenAI) to generate embeddings. It utilizes the
text-embedding-ada-002model by default. - Qdrant: A dedicated vector database that stores embeddings and associated metadata, allowing for efficient similarity searches using cosine similarity.
- LegislationService: Orchestrates the indexing of legislative articles. It breaks down laws into chunks, generates embeddings for each, and stores them in Qdrant for semantic legislation search.
- TemplateMatcher: A component of the Dynamic Template System that uses semantic search to match user questions against a library of pre-defined legal templates.
Sources: CLAUDE.md, mcp_backend/src/services/template-system/README.md, mcp_backend/src/services/legislation-service.ts
Implementation Details
Embedding Generation
The system predominantly uses OpenAI's text-embedding-ada-002 model. This model converts text strings into vectors of 1536 dimensions. These vectors are then used to calculate the "distance" or similarity between different pieces of text.
Sources: CLAUDE.md, mcp_backend/src/services/template-system/README.md
Vector Indexing Workflow
Legislative documents are indexed through a specific workflow to ensure granular retrieval (e.g., specific articles or points within a law).
| Step | Action | Description |
|---|---|---|
| 1 | Parsing | Extracts articles (статті) or points (пункти) from HTML/Text. |
| 2 | Chunking | Splits large articles into smaller segments if they exceed token limits. |
| 3 | Embedding | Calls EmbeddingService to generate vectors for each chunk. |
| 4 | Indexing | Stores the vector in Qdrant with metadata (rada_id, article_number). |
| Sources: scripts/load-bylaws.js:192-206, scripts/load-tck-legislation.js:338-348 |
Semantic Similarity Matching
In the Template System, the TemplateMatcher uses a threshold (defaulting to 0.65) to determine if a matched template is relevant enough to be presented to the user. If the similarity score falls below this threshold, the system may trigger the TemplateGenerator to create a new, more specific template.
Sources: mcp_backend/src/services/template-system/README.md
This sequence illustrates the logic used by the Template System to decide between reusing an existing template or generating a new one based on vector similarity scores.
Sources: mcp_backend/src/services/template-system/README.md
Data Structures and Configuration
Configuration Options
The vector search components are configured via environment variables to manage database connections and model selections.
| Variable | Type | Description | Default |
|---|---|---|---|
QDRANT_URL | String | URL for the Qdrant vector database. | - |
OPENAI_API_KEY | String | API key for generating embeddings. | - |
EMBEDDING_MODEL | String | The model used for vector generation. | text-embedding-ada-002 |
| Sources: CLAUDE.md, scripts/load-bylaws.js:14-15 |
Database Schema (Qdrant Metadata)
When storing vectors in Qdrant, the system includes a "payload" of metadata to allow for filtering and identification of the source text.
- rada_id: Unique identifier for the legislation (e.g.,
154-2022-п). - article_number: The specific article or point number.
- title: The title of the section.
- full_text: The raw text associated with the vector.
- legislation_id: Internal database ID for the law.
Sources: scripts/load-bylaws.js:115-131, scripts/load-tck-legislation.js:235-255
Conclusion
Vector search and embeddings are fundamental to the SecondLayer platform's ability to provide semantic legal intelligence. By leveraging OpenAI for embeddings and Qdrant for vector storage, the project enables sophisticated features like semantic legislation search, automated legal pattern analysis, and a self-learning template system. This infrastructure ensures that the AI can find relevant legal information based on conceptual similarity rather than just exact keyword matches.
Sources: CLAUDE.md, mcp_backend/src/services/template-system/README.md