Vector Embeddings & Similarity
Relevant source files
The following files were used as context for generating this wiki page:
- apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceEmbeddingServiceTest.kt
- apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceSimilarityServiceTest.kt
- apps/api/src/test/kotlin/com/briefy/api/infrastructure/enrichment/SourceEmbeddingJdbcRepositoryIT.kt
- apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt
- AGENT_NOTES.md
- apps/api/src/main/resources/db/migration/V20260219100000__source_embeddings_pgvector.sql (referenced via AGENT_NOTES.md)
Vector Embeddings and Similarity search represent the core retrieval-augmented generation (RAG) capabilities of the Briefy platform. By converting source text into high-dimensional numerical vectors, the system enables semantic search and discovery across a user's knowledge base, moving beyond simple keyword matching to understanding the underlying concepts and context of the stored information.
This system is primarily utilized for finding related content, ensuring "truth grounding" during briefing generation, and organizing information into semantic clusters. It leverages the pgvector extension in PostgreSQL to perform efficient cosine similarity searches directly within the database layer.
Sources: AGENT_NOTES.md:58-61, apps/api/src/test/kotlin/com/briefy/api/infrastructure/enrichment/SourceEmbeddingJdbcRepositoryIT.kt:47-52
Architecture and Components
The vector similarity system is organized into a clean architecture with clear boundaries between the application logic, domain definitions, and infrastructure implementation.
Key Components
| Component | Responsibility |
|---|---|
EmbeddingAdapter | Interface for communicating with external AI providers to generate vectors from text. |
SourceEmbeddingService | Orchestrates the generation of embeddings for Source entities and handles persistence logic. |
SourceSimilarityService | Provides high-level methods for querying similar sources based on text inputs. |
SourceEmbeddingRepository | Domain interface for storing and retrieving vectors. |
SourceEmbeddingJdbcRepository | Infrastructure implementation using JdbcTemplate and PostgreSQL's pgvector. |
Sources: apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceEmbeddingServiceTest.kt:21-25, apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceSimilarityServiceTest.kt:15-18
Data Flow for Embedding Generation
When a source is finalized, the system generates an embedding to make it searchable.
The system generates embeddings only for active sources. If the external embedding provider fails (e.g., due to a timeout or API error), the persistence step is skipped to prevent partial or invalid data.
Sources: apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceEmbeddingServiceTest.kt:33-52, apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt:530-536
Similarity Search Logic
The similarity search identifies semantically related sources by calculating the cosine distance between a query vector and stored source vectors.
Search Execution Flow
The SourceSimilarityService orchestrates the conversion of a raw text query into a search result set.
The search process is strictly scoped by userId to ensure data privacy. It also supports excluding specific sourceId values to avoid returning the source being currently viewed as a "similar" result.
Sources: apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceSimilarityServiceTest.kt:31-48, apps/api/src/test/kotlin/com/briefy/api/infrastructure/enrichment/SourceEmbeddingJdbcRepositoryIT.kt:56-61
Clamping and Validation
To maintain performance and reliability, the similarity service implements several guards:
- Blank Queries: Queries containing only whitespace are rejected immediately without calling the AI provider.
- Non-positive Limits: Requests for 0 or fewer results return an empty list.
- Maximum Limits: The results limit is clamped to a maximum (e.g., 50) to prevent excessive database load.
Sources: apps/api/src/test/kotlin/com/briefy/api/application/enrichment/SourceSimilarityServiceTest.kt:65-95
Infrastructure and Persistence
The implementation relies on pgvector for vector arithmetic and indexing within the PostgreSQL environment.
Database Schema (source_embeddings)
The source_embeddings table links sources to their vector representations.
| Field | Type | Description |
|---|---|---|
source_id | UUID | Primary key; FK to sources.id. |
user_id | UUID | Used for tenant-scoping queries. |
embedding | vector(1536) | The numerical representation (OpenAI standard dimensions). |
created_at | timestamp | Record creation time. |
Sources: apps/api/src/test/kotlin/com/briefy/api/infrastructure/enrichment/SourceEmbeddingJdbcRepositoryIT.kt:47-50, AGENT_NOTES.md:58-61
Implementation Details
The persistence layer uses NamedParameterJdbcTemplate. A notable technical finding is that when using PostgreSQL with JDBC, Instant types must be explicitly bound as Timestamp.from(now) to ensure correct type inference by the driver. Additionally, for integration testing, it is critical to use saveAndFlush on the Source entity before performing JDBC embedding operations to avoid foreign key violations in the same transaction.
Sources: AGENT_NOTES.md:25-28, apps/api/src/test/kotlin/com/briefy/api/infrastructure/enrichment/SourceEmbeddingJdbcRepositoryIT.kt:85-88
Summary
The Vector Embeddings and Similarity system provides the technical foundation for semantic intelligence within the repository. By integrating external AI embedding models with specialized database storage (pgvector), the system enables features like related content discovery and evidence-based synthesis. The implementation prioritizes security via strict user-scoping and system stability through input validation and transient failure handling in asynchronous embedding generation.