Embedding Model Integrations
Relevant source files
The following files were used as context for generating this wiki page:
The embedding generation system is a core component of the Technical Blog Monitor, responsible for converting extracted article text and images into numerical vectors. This process enables semantic search capabilities by storing these embeddings in a vector database. The system is designed with a pluggable architecture, allowing different embedding providers to be used interchangeably through configuration.
The primary goal is to create multimodal embeddings (text and image) for each processed blog post. These embeddings, along with associated metadata, are encapsulated in an EmbeddingRecord before being upserted into the configured vector database. The entire process is asynchronous and integrated into the main article processing pipeline.
Sources: WARP.md, monitor/main.py:257-308, README.md
Architecture and Core Components
The embedding system is built around a factory pattern that instantiates a specific embedding client based on the application's configuration. This decouples the main application logic from the details of any particular embedding provider.
Sources: monitor/main.py:93-99, monitor/embeddings/init.py, WARP.md
Factory Pattern
The get_embedding_client function acts as a factory, returning an initialized instance of an embedding client that conforms to the BaseEmbeddingClient interface. The selection is driven by the text_model_type specified in the EmbeddingConfig.
The following diagram illustrates this factory pattern.
This diagram shows how AppContext uses get_embedding_client with EmbeddingConfig to select and create a specific client implementation.
Sources: monitor/main.py:96, monitor/config.py:107-113, test_basic.py:80-85, monitor/embeddings/init.py
Core Data Model: EmbeddingRecord
The EmbeddingRecord is the primary data structure used to hold the generated embeddings and associated metadata. It is designed for eventual storage in a vector database.
The class diagram below shows the structure of the EmbeddingRecord model.
Sources: monitor/models/embedding.py:14-43
Key fields of the EmbeddingRecord are detailed below:
| Field | Type | Description |
|---|---|---|
id | str | A unique identifier for the record, typically derived from the source BlogPost. |
url | HttpUrl | The URL of the original article. |
title | str | The title of the original article. |
text_embedding | List[float] | The numerical vector representing the article's text content. |
image_embedding | Optional[List[float]] | The numerical vector representing the article's primary image or screenshot. |
publish_date | Optional[datetime] | The publication date of the article. |
metadata | Dict[str, Any] | A flexible dictionary for storing additional metadata like source, author, tags, and word count. |
Sources: monitor/models/embedding.py:14-43, monitor/main.py:290-302
Configuration
All settings for the embedding system are managed through the EmbeddingConfig Pydantic model. Configuration is loaded from environment variables with the prefix EMBEDDING__.
Sources: monitor/config.py:104-143, WARP.md
EmbeddingModelType
The choice of embedding provider is determined by the EmbeddingModelType enum.
| Enum Value | Provider |
|---|---|
OPENAI | OpenAI API |
HUGGINGFACE | HuggingFace API |
SENTENCE_TRANSFORMERS | Local Sentence Transformers models |
OLLAMA | Local Ollama instance |
CUSTOM | A dummy client for testing and development. |
Sources: monitor/config.py:96-102
EmbeddingConfig Settings
The following table details the available configuration options within the EmbeddingConfig class.
| Setting | Type | Default | Description |
|---|---|---|---|
text_model_type | EmbeddingModelType | openai | The type of embedding provider to use for text. |
text_model_name | str | text-embedding-ada-002 | The specific model name to use for text embeddings. |
image_model_type | Optional[EmbeddingModelType] | None | The type of embedding provider for images. |
image_model_name | Optional[str] | None | The specific model name for image embeddings. |
openai_api_key | Optional[SecretStr] | None | API key for OpenAI. Required if text_model_type is openai. |
huggingface_api_key | Optional[SecretStr] | None | API key for HuggingFace. Required if text_model_type is huggingface. |
embedding_dimensions | Optional[int] | None | The dimension of the text embedding vectors. |
image_embedding_dimensions | int | 512 | The dimension of the image embedding vectors. |
batch_size | int | 8 | The number of items to process in a single batch. |
max_retries | int | 3 | The maximum number of retries for a failed API call. |
timeout_seconds | int | 30 | Request timeout in seconds. |
local_model_path | Optional[Path] | None | Filesystem path to a locally stored model. |
use_gpu | bool | False | Whether to use a GPU for local models. |
Sources: monitor/config.py:104-143
Processing Flow
The embedding generation is triggered within the process_post function in monitor/main.py. After an article's content has been extracted, the embedding client is used to generate text and, if configured, image embeddings.
The following sequence diagram shows the embedding generation process for a single post.
This diagram illustrates how process_post orchestrates content extraction, screenshotting, embedding generation, and final storage in the vector database.
Sources: monitor/main.py:257-308
Implementations
Ollama Integration
The OllamaEmbeddingClient provides an integration with a locally running Ollama instance. It is selected by setting EMBEDDING__TEXT_MODEL_TYPE=ollama.
Key characteristics:
- Connection: Connects to a local Ollama API, typically at
http://localhost:11434. - API Endpoint: Sends POST requests to the
/api/embeddingsendpoint. - Payload: The request payload includes the model name, the text prompt, and a
keep_aliveparameter to keep the model loaded in memory. - Concurrency: Uses
httpx.AsyncClientwith connection limits to manage concurrent requests. - Resilience: Implements an exponential backoff retry mechanism for transient HTTP errors.
- Image Support: This client does not support image embeddings. If an image embedding is requested, it logs a warning and returns a dummy vector.
Sources: monitor/embeddings/ollama.py:25-60, 203-214
The client's core logic for embedding a single piece of text involves a POST request to the Ollama server.
# From monitor/embeddings/ollama.py:142-150
async def _embed_one(self, text: str) -> List[float]:
# ...
payload = {
"model": self.model,
"prompt": text,
"keep_alive": self.keep_alive,
}
url = f"{self.base_url}/api/embeddings"
# ...
Sources: monitor/embeddings/ollama.py:142-150
Dummy (Custom) Integration
For development and testing, a DummyEmbeddingClient is available by setting EMBEDDING__TEXT_MODEL_TYPE=custom. This client generates zero-filled vectors of a configured dimension without making any external calls, allowing for testing of the pipeline without requiring API keys or external services.
Sources: test_basic.py:77-85, monitor/config.py:101