Vector Database Integrations

Relevant source files

The following files were used as context for generating this wiki page:

• monitor/vectordb/init.py
• monitor/vectordb/pgvector.py
• monitor/config.py
• monitor/models/embedding.py
• monitor/main.py
• test_basic.py

The vector database integration is a pluggable system responsible for storing and retrieving article embeddings for semantic search. It features a modular architecture that abstracts the underlying database technology, allowing different vector databases to be used interchangeably. The system is designed around a central factory function that provides a specific client implementation based on the application's configuration.

The core data unit for all database operations is the EmbeddingRecord, which encapsulates text and image vectors along with rich metadata for each processed blog post. The primary and most complete implementation is for PostgreSQL with the pgvector extension, but stubs and an in-memory version exist for other databases like Qdrant, Chroma, and for testing purposes.

Sources: monitor/vectordb/init.py, monitor/models/embedding.py, WARP.md

Architecture and Core Components

The vector database system is built on a factory pattern that decouples the application from specific database clients. A central protocol, VectorDBClient, defines a standard interface for all interactions.

Sources: monitor/vectordb/init.py:29-166

Client Factory (get_vector_db_client)

The get_vector_db_client function in monitor/vectordb/__init__.py is the entry point for obtaining a database client. It takes a VectorDBConfig object and returns an initialized instance of the corresponding client.

This diagram illustrates the factory logic for selecting a vector database client.

VectorDBClient Protocol

All vector database clients must adhere to the VectorDBClient protocol, which ensures a consistent API for operations like storing records, searching, and counting.

This class diagram shows the VectorDBClient protocol and its relationship with the base and concrete client implementations.

Configuration (VectorDBConfig)

All vector database settings are managed through the VectorDBConfig Pydantic model. Configuration is loaded from environment variables with the prefix VECTOR_DB__.

Parameter	Type	Default	Description
db_type	VectorDBType (Enum)	qdrant	The type of vector database to use (e.g., pgvector, qdrant).
connection_string	str	http://localhost:6333	The connection string or URL for the database.
api_key	SecretStr	None	Optional API key for authentication.
collection_name	str	technical_blog_posts	The name of the collection or table to store vectors in.
text_vector_dimension	int	1536	The dimension of the text embedding vectors. Defaults to OpenAI ada-002 size.
image_vector_dimension	int	None	The dimension of the image embedding vectors.
distance_metric	str	cosine	The distance metric for similarity search (e.g., cosine, euclidean, dot).
batch_size	int	100	The number of records to process in a single batch for upsert operations.
timeout_seconds	int	30	Connection and command timeout in seconds.

Sources: monitor/config.py:161-192

Data Model (EmbeddingRecord)

The EmbeddingRecord model is the standard data structure for storing processed articles in the vector database. It contains the embedding vectors and associated metadata.

Field	Type	Description
id	str	Unique identifier for the record.
url	HttpUrl	URL of the source content.
title	str	Title of the source content.
text_embedding	List[float]	The numerical vector representing the article's text.
image_embedding	Optional[List[float]]	The numerical vector representing a screenshot of the article.
publish_date	Optional[datetime]	The publication date of the article.
metadata	Dict[str, Any]	A dictionary for storing additional, unstructured metadata.

Sources: monitor/models/embedding.py:16-51

This sequence diagram shows how an EmbeddingRecord is created and stored during the processing of a post.

PgVectorDBClient Implementation

The PgVectorDBClient is the primary, fully-featured implementation that uses a PostgreSQL database with the pgvector extension.

Sources: monitor/vectordb/pgvector.py

Initialization and Schema

On initialization, the client connects to PostgreSQL, ensures the vector extension is enabled, and creates the necessary table and indexes.

The initialize method performs the following steps:

1. Creates an asyncpg connection pool.
2. Registers the vector type with the connection.
3. Executes CREATE EXTENSION IF NOT EXISTS vector.
4. Creates the main table (e.g., blog_posts_technical_blog_posts) if it doesn't exist.
5. Creates ivfflat indexes on the text_embedding and image_embedding columns to accelerate similarity searches.

Sources: monitor/vectordb/pgvector.py:48-99

The following ER diagram shows the schema of the table created by PgVectorDBClient.

Operations

Upsert

The upsert method inserts or updates a record. It uses an INSERT ... ON CONFLICT (id) DO UPDATE statement to handle both new and existing records atomically.

-- Simplified example of the upsert query
INSERT INTO {self.table_name} (
    id, url, title, source, author, publish_date,
    text_embedding, image_embedding, metadata, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, NOW())
ON CONFLICT (id) DO UPDATE SET
    title = EXCLUDED.title,
    text_embedding = EXCLUDED.text_embedding,
    -- ... other fields
    updated_at = NOW()

Sources: monitor/vectordb/pgvector.py:126-150

Search

The search_by_text method performs a similarity search using the cosine distance operator (<=>) provided by pgvector. The score is calculated as 1 - cosine_distance to convert it to a similarity score where higher is better.

-- Simplified example of the search query
SELECT id, url, title, ...,
       1 - (text_embedding <=> $1::vector) AS score
FROM {self.table_name}
WHERE text_embedding IS NOT NULL
AND 1 - (text_embedding <=> $1::vector) >= $2 -- min_score
ORDER BY text_embedding <=> $1::vector -- Order by distance (ascending)
LIMIT $3

Sources: monitor/vectordb/pgvector.py:302-316

Other Implementations

InMemoryVectorDBClient

This client is used for testing and development. It stores EmbeddingRecord objects in a Python dictionary in memory and performs similarity calculations using numpy. It supports cosine, euclidean, and dot product metrics. This client is the default fallback if a configured database type is not fully implemented.

Sources: monitor/vectordb/init.py:277-518, test_basic.py:151-159

Stub Clients (Qdrant, Chroma, etc.)

The repository includes stub implementations for QdrantVectorDBClient and ChromaVectorDBClient. These classes exist to satisfy the factory function but do not contain real implementation logic. They log their actions and return default values. This structure makes it easy to add full support for these databases in the future.

Sources: monitor/vectordb/init.py:521-670

Summary

The vector database integration in technical-blog-monitor is a well-architected, extensible system. It uses a factory pattern and a common interface (VectorDBClient) to support multiple database backends. The configuration is centralized in VectorDBConfig, and the data is standardized via the EmbeddingRecord model. The PgVectorDBClient provides a robust, production-ready implementation, while the in-memory and stub clients facilitate testing and future expansion.