Vector Search & Embeddings
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Vector Search & Embeddings system in HEX-YT-INTEL is a core architectural component designed to transform unstructured YouTube transcripts into a structured, searchable knowledge base. By converting content into high-dimensional vector representations (embeddings), the system enables semantic discovery that goes beyond keyword matching, allowing users to query the "intelligence" extracted from videos.
This system facilitates the Knowledge System Foundation requirement of the project, ensuring that every analysis is ingestible by Retrieval-Augmented Generation (RAG) processes. It bridges the gap between raw video data and persistent knowledge graphs by utilizing Upstash for vector indexing and Supabase for metadata persistence.
Sources: web/lib/prompts/ucis-v5.1.ts:25-28, web/app/api/search/route.ts:11-15, GEMINI.md:126-130
Architecture and Data Flow
The vector search architecture follows a multi-stage pipeline: query validation, embedding generation, vector similarity search, and metadata enrichment. The system utilizes a hybrid model where vector embeddings reside in Upstash while the full analysis data is maintained in Supabase.
Semantic Search Pipeline
When a user submits a search query, the system first generates a 1536-dimensional embedding. This vector is then compared against the indexed analysis vectors using COSINE similarity.
The diagram shows the end-to-end flow from a user request to the final enriched search results.
Sources: web/app/api/search/route.ts:31-115, GEMINI.md:143-146
Component Responsibilities
| Component | Responsibility | Technical Detail |
|---|---|---|
| OpenRouter/Claude | Embedding Generation | Generates 1536-dim vectors using text-embedding-3-small. |
| Upstash Vector | Indexing & Retrieval | Stores vector IDs and metadata (e.g., analysisId). |
| Supabase | Persistence | Stores full markdown summaries, video IDs, and timestamps. |
| UCIS v5.1 | Knowledge Graph | Defines the semantic nodes and relations for RAG ingestion. |
Sources: web/app/api/search/route.ts:14-16, web/app/api/search/route.ts:147-150, web/lib/prompts/ucis-v5.1.ts:182-195
Embedding Generation
Embeddings are generated to represent the semantic meaning of both the analyzed content and user search queries. The project standardizes on 1536-dimensional vectors, optimized for compatibility with high-performance vector databases.
Generation Logic
The generateQueryEmbedding function serves as the primary gateway for converting text to vectors. It utilizes the generateEmbedding utility which interfaces with OpenRouter to access embedding models.
// web/app/api/search/route.ts:147-156
async function generateQueryEmbedding(query: string): Promise<number[]> {
try {
const result = await generateEmbedding(query);
return result.embedding;
} catch (error) {
console.error('[search] Failed to generate embedding:', error);
return [];
}
}
Sources: web/app/api/search/route.ts:147-156, web/lib/embeddings.ts
Vector Indexing and Search
Vector search is executed via the Index class from the Upstash Vector library. The system is configured to use COSINE similarity as the primary metric for determining the relevance between the query vector and stored analysis vectors.
Search Execution Flow
The system caps search results at a maximum of 50 to maintain performance. Each result returned from the vector index includes metadata that contains the analysisId used to link back to the relational database.
The sequence diagram illustrates how vector search results are mapped back to their full context in the persistence layer.
Sources: web/app/api/search/route.ts:88-115, web/lib/adapters/UpstashVectorAdapter.ts
Search API Parameters
The /api/search endpoint accepts a JSON POST request with the following schema:
| Parameter | Type | Required | Description |
|---|---|---|---|
query | string | Yes | The search term (3 to 1000 characters). |
topK | number | No | Number of results to return (default: 5, max: 50). |
Sources: web/app/api/search/route.ts:46-55
Knowledge Graph and RAG Ingestion
Vector search is tightly coupled with the Ultimate Content Intelligence & Implementation System (UCIS) v5.1. Dimension 8 of the UCIS framework specifically extracts semantic entities to build a Knowledge Graph Foundation.
Semantic Nodes and Relations
- Nodes: Domain-specific entities (People, Concepts, Frameworks, Tools) with importance weights (1-10).
- Edges: Semantic relations (e.g., "uses", "impacts", "is_part_of") with connection strengths.
- Cross-Domain Bridges: Identified connections between entirely different domains to enhance discovery pathways.
Sources: web/lib/prompts/ucis-v5.1.ts:182-205
Database Schema (Vector Search RPC)
To optimize performance, the system includes dedicated SQL functions for managing vector search operations within the relational layer.
-- supabase/migrations/20260521185646_optimize_vector_search_rpc.sql
-- Function to perform optimized vector search via RPC
CREATE OR REPLACE FUNCTION public.match_analyses (
query_embedding vector(1536),
match_threshold float,
match_count int
)
RETURNS TABLE (
id uuid,
title text,
video_id text,
similarity float
)
LANGUAGE plpgsql
...
Sources: supabase/migrations/20260521185646_optimize_vector_search_rpc.sql:1-15
Conclusion
The Vector Search & Embeddings module provides the semantic "connective tissue" for HEX-YT-INTEL. By integrating Upstash Vector indexing with Supabase metadata and the UCIS v5.1 intelligence framework, the system enables complex, persona-weighted queries across a persistent knowledge graph. This architecture ensures that even massive 2-hour masterclasses are reduced to actionable, searchable intelligence in seconds.