Adding a New Vector DB Backend

The technical blog monitor is designed with a pluggable architecture for its vector database layer. This allows developers to easily add support for new vector database backends without altering the core application logic. The system uses a factory pattern to instantiate the correct database client at runtime based on the application's configuration.

This guide outlines the architecture of the vector database abstraction and provides a step-by-step process for implementing a new backend. The primary data object handled by this layer is the EmbeddingRecord, which stores the text and image embeddings of a blog post along with its metadata.

Sources: monitor/vectordb/init.py:3-12, monitor/models/embedding.py

Architecture Overview

The vector database system is built around a few key components that ensure a consistent interface across different database implementations.

• VectorDBClient (Protocol): Defines the standard interface that all vector database clients must adhere to. It specifies the methods required for all database operations, such as upsert, search_by_text, and delete.
• BaseVectorDBClient (Abstract Base Class): Provides common functionality for all clients, including batching logic for upsert_batch and helper methods for calculating similarity scores. New clients should inherit from this class.
• VectorDBConfig (Pydantic Model): A configuration model that holds all settings related to the vector database, such as the DB type, connection string, and collection name.
• VectorDBType (Enum): An enumeration of all supported vector database types. A new backend must be added to this enum.
• get_vector_db_client (Factory Function): This function is responsible for creating and initializing the appropriate vector database client based on the VectorDBType specified in the configuration.

Sources: monitor/vectordb/init.py:15-24, 203-215, monitor/config.py:141-147, 150-175

Component Relationship Diagram

This class diagram illustrates the relationship between the core components of the vector DB layer.

Implementation Steps

To add a new vector database backend, follow these steps.

1. Update Configuration Models

First, add your new database type to the VectorDBType enum in monitor/config.py.

# monitor/config.py

class VectorDBType(str, Enum):
    """Types of vector databases supported."""
    QDRANT = "qdrant"
    CHROMA = "chroma"
    PINECONE = "pinecone"
    MILVUS = "milvus"
    WEAVIATE = "weaviate"
    PGVECTOR = "pgvector"
    YOUR_NEW_DB = "your_new_db" # Add your new type here

Sources: monitor/config.py:141-147

The VectorDBConfig model defines the configuration parameters for all vector databases. If your new backend requires specific configuration fields, you can add them as optional fields to this model.

Sources: monitor/config.py:150-175

2. Create the Client Class

Create a new Python file in the monitor/vectordb/ directory (e.g., your_new_db.py). In this file, define a new class that inherits from BaseVectorDBClient. This class must implement all the abstract methods defined by the VectorDBClient protocol.

# monitor/vectordb/your_new_db.py
from monitor.config import VectorDBConfig
from monitor.vectordb import BaseVectorDBClient
from monitor.models.embedding import EmbeddingRecord

class YourNewClient(BaseVectorDBClient):
    def __init__(self, config: VectorDBConfig):
        super().__init__(config)
        # Add any client-specific initialization here

    async def initialize(self) -> None:
        # Connect to the DB, create collections/tables/indexes
        ...

    async def close(self) -> None:
        # Close connections and clean up resources
        ...

    async def _upsert_batch(self, records: List[EmbeddingRecord]) -> bool:
        # Implement the logic to insert/update a batch of records
        ...

    # ... implement all other required methods ...

Sources: monitor/vectordb/init.py:203-215, monitor/vectordb/pgvector.py:20-36

3. Implement the VectorDBClient Interface

Your new client class must provide concrete implementations for the methods defined in the VectorDBClient protocol. The BaseVectorDBClient provides a default implementation for upsert_batch which calls _upsert_batch in a loop, so you only need to implement the latter for batching.

Sources: monitor/vectordb/init.py:18-142

The data object you will be working with is the EmbeddingRecord.

Sources: monitor/models/embedding.py:15-46

4. Update the Factory Function

Finally, update the get_vector_db_client factory function in monitor/vectordb/__init__.py to include your new client.

This diagram shows the logic flow of the factory function.

Add a new condition to the if/elif block to handle your new VectorDBType.

# monitor/vectordb/__init__.py

async def get_vector_db_client(config: VectorDBConfig) -> VectorDBClient:
    """
    Get a vector database client based on configuration.
    ...
    """
    if config.db_type == VectorDBType.PGVECTOR:
        from monitor.vectordb.pgvector import PgVectorDBClient
        client = PgVectorDBClient(config)
    
    elif config.db_type == VectorDBType.YOUR_NEW_DB: # Add this block
        from monitor.vectordb.your_new_db import YourNewClient
        client = YourNewClient(config)

    elif config.db_type == VectorDBType.QDRANT:
        client = QdrantVectorDBClient(config)
    
    # ... other types
    
    else:
        # Default to in-memory client for testing
        client = InMemoryVectorDBClient(config)
    
    # Initialize the client
    await client.initialize()
    
    return client

Sources: monitor/vectordb/init.py:808-861

Application Integration

The vector database client is initialized and managed within the AppContext class. The get_vector_db_client factory is called during application startup, and the resulting client instance is made available to the rest of the application.

This sequence diagram shows how a new blog post embedding is processed and stored in the vector database.

By following the VectorDBClient protocol and integrating via the factory function, your new backend will seamlessly plug into the existing application flow for storing and searching blog post embeddings.